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ABSTRACT 



The Oregon Department of Education (ODE) operates under two 
main statutory charges: to administer state and federal programs, and to 
serve as the state's lead agent in implementing the Oregon Educational Act 
for the 21st Century. This report presents the results of a contracted 
program evaluation of the ODE. Data were collected through document analysis; 
site visits; a telephone survey of 312 local education officials; and 
interviews with a total of 150 state-level policymakers, ODE officials, 
representatives of state agencies and interest groups, district and school 
administrators, and teachers. The study evaluated ODE on the basis of the 
kinds of activities most commonly carried out by state departments of 
education in an era of standards-based reform--leadership, capacity building, 
research and analysis, communication, and monitoring and enforcement. The 
report concludes that ODE has worked diligently to make progress in 
implementing the Oregon Educational Act for the 21st Century, but that a 
midcourse correction is needed to keep the reform on track. Specifically, the 
ODE needs to assume a more activist leadership role. With regard to the 
supervision, management, and control of schools, ODE has not adequately 
adjusted to the demands of a new policy environment for public education in 
Oregon, particularly in terms of how it monitors school finance. The report 
offers 14 recommendations. Appendices contain a description of evaluation 
methodology, an annotated bibliography of 18 sources, and a list of 
exceptions to compliance with generally accepted government auditing 
standards. (LMI) 
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This report presents the results of our contracted program evaluation of the 
Oregon Department of Education (ODE) by Management Analysis and Planning 
Associates, L.L.C. This review, which was approved by the Joint Legislative Audit 
Committee, was planned and managed by the Legislative Fiscal Office, the Department 
of Administrative Services, and the Oregon Audits Division in a cooperative effort. In 
addition, the Oregon Audits Division received input from the Department of Justice, 
which advised us that this audit could be conducted under the authority of 
ORS 297.210. 

The report concludes that ODE has worked diligently to make progress in 
implementing the Oregon Educational Act for the 21st Century, but a mid-course 
correction is in order to keep the reform effort on track. Specifically, ODE needs to 
assume a more activist leadership role in improving the quality of teaching and learning 
in Oregon’s schools. With regard to ODE’s supervision, management, and control of 
schools, the report notes that ODE has not adequately adjusted to the demands of a new 
policy environment for public education in Oregon, particularly in terms of how it 
monitors school finance. 
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To assist ODE in its educational reform efforts and to improve ODE operations, 
the report makes fourteen recommendations in the areas of leadership, monitoring and 
enforcement, capacity building, research and analysis, and communication. 
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EXECUTIVE SUMMARY 



The Oregon Department of Education (ODE) operates under two main 
statutory charges. One is to serve as the principal administrative agency for the 
State Board of Education: 

...(ODE) shall exercise all administrative functions of the state 
relating to supervision, management and control of schools 
and community colleges not conferred by law on some other 
agency. (ORS 326.111(3) 

This is the traditional regulatory and oversight role, mainly administering state 
and federal grant programs, that State Departments of Education across the 
country were originally established to discharge. 

The second role is to serve as the State's lead agent in implementing 
Oregon's ambitious school reform legislation, the Oregon Educational Act for the 
21 st Century: 

The Department of Education shall be responsible for 
coordinating research, planning and public discussion so that 
activities necessary to the implementation of this chapter can 
be achieved. Actions by the department to fulfill this 
responsibility and to increase student achievement may 
include. . . (etc.) (ORS 329.075(2)). 



With regard to the latter responsibility, this program evaluation concludes 
that ODE has worked diligently to make progress in implementing the Oregon 
Educational Act for the 21 st Century but a mid-course correction is in order to 
keep the reform effort on track. Specifically, ODE needs to assume a more activist 
leadership role in improving the quality of teaching and learning in Oregon's 
schools. With regard to the former responsibility - the supervision, management 
and control of schools - ODE has not adequately adjusted to the demands of a 
new policy environment for public education in Oregon, particularly in terms of 
how it monitors school finance. Oregon voters in 1990 lowered property taxes and 
required that the state General Fund become the major revenue resource for local 
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schools and community college districts. The share of state funding of the general 
operating revenue of local schools has increased from 27 percent in 1989-91 to 66 
percent in 1995-97. Clearly, such a preponderant state contribution implies an 
increased fiduciary responsibility on the part of state government to ensure that 
taxpayer money is being spent wisely. ODE must be in a position to provide the 
requisite data and analysis to ensure good use of state education dollars. 

Background for this Study 

This report is the culmination of seven months of review, research, and 
analysis by Management Analysis and Planning (MAP), a San Francisco-based 
education consulting firm. MAP, the successful bidder on a Request for Proposal 
released by the State of Oregon's Legislative Fiscal Office, Department of 
Administrative Services, and Secretary of State Audits Division, was asked to 
review ODE's "oversight of the local public (K-12) education system." In order to 
gather the most significant and comprehensive data, MAP employed a multi- 
pronged research strategy which included document analysis; state level 
interviews; Education Service District (ESD), school district, and school level 
interviews; and a telephone survey of a scientifically selected sample of 
approximately 300 local education officials. 

MAP looked at ODE through five lenses - leadership, capacity building, 
research and analysis, communication, and monitoring and enforcement - which 
mirror the kinds of activities that modem State Departments of Education are 
undertaking in the era of standards-based school reform. The full report's 
findings, supporting evidence and recommendations are presented in terms of 
these five organizing categories. In this executive summary, however, we have 
aggregated findings and salient evidence around the two statutory responsibilities 
described above in order to present conclusions in a digested form but in a way 



which retains their coherence. All of the report's 14 recommendations are then 
presented, with special attention to the three that would require legislative action 
for their realization. 

Findings Regarding Implementation of the 
Oregon Educational Act for the 21 st Century 

A) ODE has made solid progress in implementing the performance standards 
feature of the school reform legislation. 

Evidence in the report supporting this finding includes the fact that ODE 
has established a statewide consensus around new content and performance 
standards by specific core subject areas. These have been favorably reviewed by 
an independent, nationally respected team of educators. ODE has also developed 
statewide assessments in reading, writing and mathematics which, informed 
observers agree, authentically measure student progress in these curricular areas 
and which are tied to the awarding of the Certificate of Initial Mastery in the 10th 
grade. According to curriculum association interviews, assessment progress in the 
remaining subject areas - science and social studies - may be more problematic. 
Nevertheless, in the survey conducted for this evaluation, Oregon educators 
tended to give ODE highest approval ratings for its work establishing 
performance standards. 

B) ODE employees do not regard themselves as having an important role in 
improving teaching and learning. 

Results on statewide tests in reading, writing and mathematics in Oregon 
have been generally static for the last few years. It is widely recognized that at 
present levels of performance only one in four students will satisfy proposed state 
mathematics standards for earning the Certificate of Initial Mastery by 10 th grade. 
Nevertheless, ODE does not have a contingency plan in place for dealing with this 
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forseeable crisis nor a comprehensive implementation strategy in place to prevent 
it from happening, such as building the capacity of local districts and schools to 
meet the ambitious academic goals of the reform legislation. Pointing to the 
reform law, top ODE managers report that the responsibility for lagging student 
performance rests solely on the shoulders of local educators. The survey confirms 
that local educators do not see ODE as a resource providing technical assistance in 
such areas as professional development or financial management. In short, ODE 
seems not to have grasped that the reform statute provides only the basic blue line 
drawings for education improvement and that it remains ODE's responsibility to 
initiate a coherent set of strategies designed to bring it to fruition. 

C) ODE's current organizational structure neither conveys nor facilitates an 
activist leadership role in improving teaching and learning. 

An examination of ODE's organizational structure points to its funding 
sources, not a focus on the essentials of education - how teachers teach and what 
and how students learn. For example, the critical activity of statewide assessment 
is given equal organizational weight with special education while the various 
subject areas are nowhere to be found. In order to live within its budget, ODE has 
divested itself of its subject matter experts through reassignment or attrition. This 
is particularly problematic in Oregon because the state's largest school districts 
have also cut back on curricular staffing. Interviews revealed that ODE's 
organizational structure and intense workload discourage interstaff 
communication. Communication between ODE and the various subject matter 
associations - a vastly underutilized resource - were judged inadequate by 
representatives of those organizations. 

D) Current conditions preclude ODE from securing as employees the broad 
range of educational experts necessary to carry out the education reform. 
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Salaries for education professionals at ODE lag significantly behind those in 
Oregon school districts. As a result, ODE shows many symptoms of not being 
competitive in the job market. Comparing Oregon's level of expenditure on its 
State Department of Education with six states of comparable size, MAP 
determined that Oregon spends about eight dollars per pupil below the average of 
comparison states. If ODE were funded at the average of this group, its budget 
would be approximately $4 million larger. Moreover, 

separating out additions due to absorption of the juvenile corrections education 
program, ODE full time equivalent positions have declined from 237 full time 
equivalent (FTE) positions in 1990 when the reform law was passed to 210 in the 
current biennium. 

E) Despite efforts to move from regulation to assistance, ODE continues to be 
perceived by its clients as primarily a regulatory and compliance 
organization. 

In the survey administered for this study, the word most frequently chosen 
by local educators to describe ODE was "bureaucratic." Local educators were 
nearly unanimous in saying that they would prefer ODE to act as a coach in 
Oregon education; in point of fact, however, the largest group saw ODE acting as 
a referee. Part of the problem is statutory. For example, ODE is required by law 
to conduct school improvement visits in every district in the state once every three 
years. For a variety of reasons, MAP strongly questions whether this is an 
effective use of ODE's limited resources. 
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Findings Relating to ODE's Supervision, Management 
and Control of Schools 

F) ODE has little capacity to conduct research and analyze data on key issues 
such as student achievement and school finance. 

Interviews with policy makers indicated that ODE is a spectator in 
Oregon's school funding process, a perception that ODE managers readily 
affirmed. But the change to a predominantly state-funded and performance-based 
school system means that policy makers have a greater need than ever before for 
objective, insightful analysis on key issues. For a variety of reasons, ODE is ill- 
equipped to provide such analysis, which could lead to unfortunate outcomes. 

For example, MAP found that ODE lacks the capacity to identify school districts in 
financial trouble and does not have a contingency plan for early intervention to 
prevent potential district bankruptcies. 

G) ODE's current data collection system provides inadequate school level data. 

Current accounting rules are sufficiently ambiguous and the level of 
aggregation of data reported to ODE is sufficiently gross that reasoned judgments 
about expenditure patterns between and among school districts is virtually 
impossible. ODE collects no data, for example, that would permit analysis of 
school level expenditures. But without such data, judgments about the cost- 
effectiveness of specific interventions will not be possible. 
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H) ODE's current Management Information System is hobbled by interlocking 
conditions that compromise its utility. 

ODE's MIS lacks a coherent philosophical underpinning; collects largely 
compliance-oriented information; and has not been recalibrated to satisfy the 
changing needs of state policy makers. 

I) Annual audits represent an underutilized opportunity for appropriate state 
oversight of district expenditure patterns. 

District financial statements are currently audited annually, as required by 
the Municipal Audit Law. Although the State Auditor sets certain standards for 
audits, individual audits are not standardized in Oregon and school districts have 
little incentive to expand their scope. Many states make far better use of the 
auditing function as a cost effective alternative to operating without necessary 
information or conducting School Improvement Visits to verify compliance items. 

J) ODE does not employ systematic comprehensive planning and budgeting as 
an internal management tool. 

Interviews found that ODE managers have only provisional control over 
their office budgets. Planning is episodic and ad hoc. Priority activities may be 
planned; but one gets the impression that priorities tend to shift and proliferate. 
The Superintendent confirmed that virtually all budget decisions are made in her 
office. 
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K) The elected office of Superintendent of Public Instruction tends to fragment 
responsibility for education. 

The Oregon Constitution requires that the state superintendent be elected. 
However, real authority for determining levels of funding for education in Oregon 
resides with the governor and legislature. When programmatic leadership and 
budgeting power are split, no one is ultimately responsible to the voters for the 
functioning of the schools. 



Recommendations 



1 . ODE should assume a more activist role in improving curriculum and 
instruction in Oregon's schools. 

To do so, the full report explains, ODE needs to articulate a clear vision of 
what works in education; develop in its organization a clear understanding of the 
change process; develop a comprehensive, evolutionary implementation strategy; 
and, appreciate that education improvement is a team effort. 

Cost estimate: neutral 

2. ODE should be reorganized to reflect a priority for those functions most 
closely related to improving educational programs as well as to enhance 
necessary communication among key program improvement functions. 

The full report presents one possible model of such a restructuring. It 
recommends that ODE create the position of Chief Deputy Superintendent with 
lead responsibility for improving teaching and learning in local schools. It also 
recommends that ODE rebuild its subject matter expertise. 

Cost estimate: $100,000-125,000 for Chief Deputy Superintendent 

3. ODE should develop for every organizational unit annual work plans with 
measurable outcomes and budgets specified. 

Cost estimate: neutral 

4. ODE, in cooperation with the legislature and appropriate state agencies, 
should take the steps necessary to attract and retain professionals with 
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sufficient credibility to lead implementation of Oregon's Educational Act for 
the 21 st Century. 

Cost estimate: Deputy Superintendents should be paid $90-95,000 per year; 

Assistant and Associate Superintendents $75-80,000; Specialists, $60-75,000. 

5. The elected position of Superintendent of Public Instruction should be 
replaced with an appointed position. 

Most states have turned to this solution - only 15 still elect their chief state 
school officer. A strong majority of educators, according to the survey, favor such 
a change. 

Cost estimate: $125-150,000 less present SPI's salary 

6. Conventional kinds of monitoring and enforcement activities should be 
streamlined and reduced. 

For example, the law requiring ODE to do School Improvement Visits 
should be repealed. Cost effective alternatives are suggested in the full report. 

Cost estimate: Unspecified savings should be used by ODE for more productive 

ends. 

7. Essential monitoring and enforcement activities should be consolidated into 
a single, relatively small unit of ODE. 

Cost estimate: Unspecified savings 

8. ODE should strategically redeploy the resources currently being used for 
"technical assistance" in Oregon to create multiple networks of service 
providers across the state. ODE's role should be as a catalyst and 
clearinghouse for such efforts, not as a direct provider. 

See Chapter Five, "Perspectives on Implementation" for an example of how 
another state has handled this challenge. 

Cost estimate: $2-3 million for 5000 teachers in Oregon going through 3-5 week 

summer workshops each summer with follow-up exercises through the year. 

9. ODE should assign a much higher priority to the professional development 
of its own staff. 

Cost estimate: neutral, but time must be scheduled for this important activity. 



10. ODE needs to greatly increase its research and analytical capability. 

Cost estimate: Approximately $150,000 - 225,000 for 2-3 FTE with expertise in 
finance, assessment and data management. 

11. ODE needs to play a more active role in the development, collection and 
reporting of financial information. 

The full report recommends that ODE assign its highest priority to 
standardizing accounting procedures among districts. It recommends that ODE 
analyze district budgets before the beginning of each school year and require at 
least one mid-year expenditure report. It also recommends that ODE develop 
contingency plans for dealing with districts which find themselves nearing 
bankruptcy and suggests a model intervention plan from another state with 
predominantly state-funded schools. 

Cost estimate: $2-3 million to create a uniform data system; plus approximately 
$120,000 - 180,000 for 2-3 FTE to increase analytic capability. 



12. ODE should establish a single departmental database of information that is 
accessible to anyone within ODE and that facilitates responsive answers to 
policy makers and other interested parties. 

The full report recommends that ODE must first: determine what kind of 
data policy makers and others might want; second, develop a "data dictionary" 
(directory of definitions and locations); third, determine the frequency and mode 
of data collection; and, fourth, move toward a standardized format for all data. 

Cost estimate: ODE plans to use hardware from another agency; software costs 
are already in ODE's budget. Training costs will be incurred but only for those 
individuals with direct responsibility for operating the data system. 

13. ODE in conjunction with the State Auditor should expand existing local 
district CPA audits to provide more information for policy makers. 

Cost estimate: Costs need to be calculated based on the extent to which audit 
scope is expanded. 



14. ODE should adjust its communications to its various "publics" and develop 
feedback loops capable of measuring the effectiveness of its efforts. 

Cost estimate: $50-75,000 to perform annual surveys. 
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Chapter 1 



INTRODUCTION 



This report represents an evaluation of the Oregon Department of 
Education (ODE). It is the culmination of seven months of review, research, and 
analysis by Management Analysis and Planning (MAP) Associates, a California- 
based education consulting firm. MAP, the successful bidder on a Request for 
Proposal (RFP) released by the State of Oregon's Legislative Fiscal Office, 
Department of Administrative Services, and Secretary of State Audits Division 
was asked to review ODE's "oversight of the local public (K-12) education 
system." The scope of this appraisal, in other words, is the role and function of the 
Oregon Department of Education vis a vis the state's kindergarten through twelfth 
grade system of public schools. 

BACKGROUND ABOUT ODE 

The Oregon Department of Education is the State's lead agency for 
overseeing and improving Oregon's public prekindergarten through grade 12 and 
community college education systems. ODE's mission, as directed by the State 
Board of Education, is to assure excellent and equitable educational opportunities 
resulting in the development of every Oregonian's self-esteem, potential 
(including academic skills and knowledge), ability to enter the workforce, and 
lifelong learning capability. ODE is headed by the State Superintendent of Public 
Instruction, an elected position established by the Oregon Constitution. The 
statutorily-authorized State Board of Education is responsible for policymaking, 
planning, and evaluation of public elementary, secondary and community college 
education. A major role of the Superintendent, State Board and ODE in recent 



years has been to provide statewide leadership in the implementation of Oregon's 
sweeping, standards-based education reform legislation, the Oregon Educational 
Act for the 21 st Century. 

ODE serves 220 elementary and secondary school districts and 23 education 
service districts (ESDs), which in turn serve some 552,000 elementary and 
secondary school students (grades K-12). In addition, ODE directly manages the 
Oregon School for the Blind, the Oregon School for the Deaf, and most of the 
education programs for adjudicated youth. In carrying out its responsibilities, 
ODE interacts with a number of associated state entities including the Teacher 
Standards and Practices Commission, Office of Community College Services, 
Oregon Progress Board, Workforce Quality Council, Oregon State System of 
Education, Oregon Commission on Children and Families, and the Oregon Youth 
Authority. 

ODE's budget for the 1995-97 biennium (see Tables which follow) totals 
$4.15 billion, of which $3.55 billion is distributed as state support to local school 
districts, ESDs, and juvenile corrections. An additional $514.7 million, including 
federal funds, passes through to local districts in the form of grants-in-aid for 
specific special education, pre-kindergarten, nutrition and other programs. ODE's 
adopted operating budget, not including the cost of running the various special 
schools listed above, is $44.4 million, which funds 209.63 full-time equivalent 
employees to carry out ODE's administrative and program management 
responsibilities. 
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ODE OPERATING BUDGET 
by expenditure category and fund source 
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ODE TOTAL BUDGET 
by expenditure category and fund source 
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In the wake of voter-approved property tax reform, the State share of 
funding of the general operating revenue of local schools has increased from 27 
percent in 1989-91 to 66 percent in 1995-97. As a result, the total amount of money 
passing through ODE has increased tremendously in recent years - from $2.3 
billion in 1991-93 to $4.15 billion in 1995-97. At the same time, however, ODE's 
operating budget has remained relatively static or even declined in absolute terms 
- going from $44.67 million in 1993-95 to $44.4 million in 1995-97. 

ODE has a relatively flat organizational structure. Currently, two deputy 
superintendents and seven assistant or associate superintendents report directly to 
the Superintendent of Public Instruction. Organizational units reporting directly 
to the Superintendent include Special Education; Professional Technical 
Education; Student Services; Compensatory Education; Curriculum, Instruction 
and Field Services; Assessment; State Board Relations; and Education Support 
Services. 

Additional detail on management structures, procedures, and techniques 
can be found in Chapter 3 Findings of this report. 

STATE POLICY CONTEXT 

During the 1990s, the governor and Legislative Assembly (legislature) 
have come to play an increasingly prominent role in education matters in 
Oregon. While it may be the case, as one official told us, that, "Oregon is no place 
to practice centralized government," it is also true that over the course of the last 
decade, the State — in the form of the governor and the legislature— has come to 
occupy a larger place at the education table. This new, and more prominent, state 
role in education matters is entirely appropriate. 

Education, to be sure, is important to individual citizens' well-being. 
Increasingly, however, education is also important to the health of Oregon's 
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economy. Oregon is no longer a loosely coupled confederation of independent, 
and largely self-sufficient, agrarian communities. The State is ever more reliant 
on, for example, firms that specialize in high tech, draw workers from throughout 
Oregon, and require employees with solid educational backgrounds and high 
levels of skills. A robust economy requires that state policy makers be concerned 
with ensuring that the State's young people adequately are prepared for jobs in 
these new Oregon industries. 

In addition to these economic concerns, two critical events— the passage by 
Oregon citizens of tax limitation measures and the enactment by the legislature of 
a major education reform statute— have put state policy makers front and center 
in education decision making. 

While on many dimensions Oregon still retains its strong historical 
commitment to local control, the implementation of tax limitation, as specified in 
Measure 5, has transformed the education finance system into one in which the 
state dominates. The state collects and distributes dollars to local school districts. 
State lawmakers, rather than local school boards, now have a fiduciary 
responsibility to Oregon taxpayers to ensure that those dollars are well spent. 

The Educational Act for the 21 st Century, Oregonian omnibus reform 
measure, has put the State in the driver's seat in terms of "drawing the box" 
around what is expected of schools and required of students. While local districts 
and schools are expected to determine how student achievement will be 
improved, the legislature now specifies what districts and schools must focus on 
and the standards by which their accomplishments will be measured. 

The current governor has given education a prominent place on his 
agenda. He has appointed a blue ribbon task force to identify the components of 
a "basic education" and to determine the cost of ensuring that all students in 
Oregon are able to gain access to such an education. More recently the governor 
has proposed a statewide teacher salary schedule and a form of "pay for 
performance" which would monetarily reward groups of teachers whose students 
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achieve at rates faster than anticipated. MAP's purpose here is not to comment 
on the merits of any of these initiatives, but simply to use them as illustrations of 
the governor's interest in and commitment to education as a key policy matter for 
the State. 

The imperatives of a changing economy, citizen-enacted tax limitation, and 
a statewide reform initiative, then, all have combined to give state policy makers 
an ever larger stake in education policy and increasing opportunities, and 
obligations, to exert education leadership. However, the state legislature has sent 
mixed messages regarding its commitment to education reform. 

The Educational Act for the 21 st Century was passed in 1991, amended in 
1995, and will be up for review before a legislative committee in 1997. While 
some amount of mid-course correction in a reform effort as ambitious as this one 
is to be expected, the legislature is widely perceived as continually modifying— 
or threatening to modify— the essential purpose and substance of the Act, thus 
engendering skepticism regarding the State's commitment to education reform. 
This chary view of the State's intention to continue for the foreseeable future 
down the path it has laid out was captured in the statement of one official who 
reported that, "Reality [in terms of education] exists in Oregon in two-year 
increments," an obvious reference to the legislative biennium. 

The statutorily-authorized State Board of Education is not generally viewed 
as a significant player in Oregon's education policy arena. State Board members 
clearly are committed to their responsibilities; they take them seriously and devote 
much time and effort to state board work. Nonetheless, the consensus among 
legislators, representatives of major interest groups, and local educators is 
consistent: the State Board of Education wields little influence over matters of 
education policy. 

We hasten here to add that the role Oregon's State Board of Education has 
carved out is appropriate. A state board of education should set policy, within the 
context of state law. The department of education then has the responsibility to 
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administer that policy. A state board of education should not— and Oregon's does 
not— compete with the governor and legislature to establish board-scale macro- 
policy, nor should the state board involve itself with the details of legislative 
implementation. 

In many states, court actions have shaped education policy. Desegregation 
decisions have determined district and school boundary lines and means for 
assigning students to schools. Finance decisions have prescribed the methods by 
which dollars are allocated. More far-reaching judicial determinations in some 
states have actually provided the architecture for education reform policies and 
practices. 

This is not the case in Oregon. There is no significant education area in 
which litigation or other forms of judicial action have been the driving forces. To 
be sure, legal action is sometimes threatened and lawsuits occasionally filed, but at 
this juncture, neither threats nor actual suits have had a direct impact on 
fundamental matters of state education policy or practice. 

Various interest groups— those representing administrators, school boards, 
teachers and parents, for example— are influential at the state level on individual 
and particular education matters. However, it is rare for interest groups to 
coalesce and speak with one voice on issues. While each group holds opinions on 
matters central to education in Oregon, there is no evidence that these 
organizations function as a policy coalition. Thus, on balance, interest group 
influence over education policy matters is rather diffuse. 

MAP’S FRAME OF REFERENCE 

Before briefly describing the methodology employed in the conduct of this 
study and outlining the organization of the remainder of this report, we want to 
emphasize the criticality of context for the findings and results of this work. By 
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context we mean, in particular, the central role education and education reform 
have come to play in Oregon. 

Implementing the Educational Act for the 21 st Century is the dominant 
function of ODE. It is this responsibility — transforming a comprehensive statute 
enacted by Oregon's elected state leaders into a plan of operation that will enable 
districts and schools to realize the Act's promise of improved student 
achievement— that has swamped all other ODE tasks. And rightly so. 
Implementing reform on a grand scale is complex, cumbersome, and time 
consuming. It requires the agency to reconceptualize its mission, rethink its 
priorities and reorient its staff. It requires, in short, ODE to focus consistently and 
systematically on both the "big picture" and the details of the reform law. It is the 
reform act that, in large measure, now gives shape and substance to ODE's role 
and its responsibilities. 1 

Why does all of this matter? It matters because in thinking about how ODE 
goes about doing its work, the requirements of improving teaching and learning 
are paramount. Principal ODE alignments and operations need to be constructed 
around this obligation of enhancing education quality and improving student 
achievement results. 

To illustrate more concretely, let us consider for a moment organizational 
structure. A reasonable person might well assume that there exists some generic 
management structure that successful state departments of education ought to 
emulate. After all, is there not some sort of template, specifying numbers and 
types of personnel and assigned duties, that could be superimposed on any state 
agency charged with responsibility for education? 

The answer is "no." Organization and management structures are 
contextual. The "test" for ODE is not the number of accountants or deputy 
superintendents or program area specialists it employs. The test is the extent to 



1 The only other defining dynamic in the Oregon education arena which carries such an impact is tax 
limitation and the shift to a state-funded school system. 



which ODE's organizational structure enhances or inhibits the agency's ability to 
carry out, and to assist districts and schools to carry out, the purposes of the 
Educational Act for the 21 st Century. No amount of legislative manipulation or 
infusion of new management procedures or techniques can compensate for 
inadequate vision and leadership. The challenges before ODE are educational 
challenges, not generic management issues. 

What, then, a reader might ask, would happen if the reform act was 
replaced by an entirely different statute, one that specified a different set of duties 
and responsibilities? Would the same organizational structure that serves ODE 
well when one statute is dominant serve equally well when another prevails? The 
answer is, "probably not." 

An agency must be organized for some specific purpose or set of purposes, 
and must be managed to accomplish a particular set of goals. Alter the purposes 
and goals and the structure must also be modified. 

SCOPE AND METHODOLOGY 

The scope of this evaluation is ODE's role and function in relation to the 
state's kindergarten through twelfth grade public school system. Specifically 
excluded were the state special schools for the deaf and blind and the Office of 
Community College Services. 

In order to gather the most comprehensive and accurate data, MAP 
employed a multi-pronged research strategy which included document analysis, 
interviews, and a telephone survey of a scientifically selected sample of local 
education officials. In the first phase of the evaluation, MAP identified the critical 
issues to be explored, developed provisional study questions, reviewed salient 
background documents, and conducted preliminary interviews with key ODE 
officials and state policy makers. 
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Next, MAP refined its workplan, adjusted study questions, and generated 
questions for the survey of local educators. The third phase of the study consisted 
of fact-finding and analysis. During this period, MAP engaged in additional 
document review, conducted numerous interviews, and administered and 
analyzed the results of the previously mentioned statewide survey. These analytic 
activities led to the validation, and sometimes the rejection, of working 
hypotheses. 

Phase four consisted of identifying findings and forming conclusions and 
recommendations. These were "tested" with ODE officials, other state officials, 
and with the external advisory board which provided advice and counsel to MAP 
during the course of this study. 

We reviewed audit workpapers prepared by the Audits Division for ODE 
for the 1996 statewide financial audit including the state school support 
apportionment formula. We also reviewed the department's budget. 

Finally, a report was prepared and submitted to the client. Except as noted 
in Appendix C, this audit was conducted in accordance with generally accepted 
government auditing standards. We limited our review to the areas specified in 
this section of the report. 



ORGANIZATION OF THIS REPORT 



This report is divided into six chapters: 

• Chapter 1 is the introductory material contained here. 

• Chapter 2 more fully describes evolving roles for departments of education 
generally and for the Oregon Department of Education specifically. 

• Chapter 3 is the set of Findings, in other words, the data-driven conclusions 
derived from MAP's study of ODE. Findings, which are numbered 
sequentially for ease of reading, are divided into five categories: 1) leadership, 
2) monitoring and enforcement, 3) capacity building, 4) research and analysis, 
and 5) communication. 

• Chapter 4 presents MAP's Recommendations regarding changes ODE might 
consider in order to improve its efficiency and effectiveness. 

Recommendations are also numbered sequentially and divided into the five 
previously described categories. 

• Chapter 5 presents the results of a survey of local Oregon educators conducted 
for this study. 

• Chapter 6, Perspectives On Implementation, offers a set of lessons learned and 
exemplary practices from other states that are undertaking reform efforts 
similar to Oregon's. 

• Chapter 7 offers some concluding remarks. 

Following the five chapters are two supporting appendices: 

• Appendix A describes MAP's methodology for this study —how we obtained 
and analyzed data that allowed us to generate findings and develop 
recommendations. 

• Appendix B provides a set of references. It is, in effect, a "Read More About It" 
section that offers suggested books, papers, and reports for those interested in 
pursuing more deeply some of the topics germane to this study. 
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Chapter 2 



THE CONTEXT FOR CHANGE 



This report begins from a relatively simple premise: that the mission of a 
state department of education is, or ought to be, continuous improvement of 
teaching and learning in local schools. In Oregon the Legislative Assembly has 
made this responsibility explicit, assigning the Oregon Department of Education a 
leadership role in K-12 school improvement as outlined in the Oregon Educational 
Act for the 21 st Century. Taken together, the 1991 reform law (HB 3565) and its 
1995 companion, HB 2991, have created an ambitious set of expectations by subject 
area and age group for Oregon students to attain. Progress toward meeting these 
goals is one index of the relative effectiveness of ODE. Hence, the title of this 
report: "To Lead Education Reform: An Appraisal of the Oregon Department of 
Education." 

At the same time the state is engaged in its omnibus education reform 
efforts, Oregon has undergone a fundamental shift in the way in which it finances 
its schools. Voter-approved initiatives have transformed Oregon's school finance 
system from one that was property-tax-based and locally controlled to a system in 
which the state is the principal source of education revenue. Thus, state policy 
makers are at once concerned about the results of the education reform they have 
set in motion and more intensely interested in the dollars they now distribute to 
districts and schools. 

Substantive improvement in Oregon's schools is certainly a policy goal 
devoutly to be wished. Unfortunately, as Sam Johnson put it, "Change is not 
made without inconvenience, even from worse to better." For better or worse, 
Oregonians have been engaged in a persistent effort to reinvent their public school 
system over at least the last twenty years. This Chapter reviews the larger state 
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and national context for this effort. It reviews, first, the underlying forces that 
have driven change in Oregon's public school system; second, some reasons why 
school improvement consistently turns out to be far more difficult to accomplish 
in practice than on paper; and, third, the nature of the new leadership role that 
State Departments of Education (SDEs) have begun to take on as standards-based 
reform unfolds across the nation. 

THE CHANGING LANDSCAPE OF EDUCATION REFORM 

Oregon has a reputation for finding original solutions to tough policy 
challenges. The Bottle Bill, public access to beaches, and the Oregon Health Plan 
are all examples of innovations with a "made in Oregon" stamp that have 
attracted widespread national attention and, not infrequently, imitation. So it is 
perhaps not surprising that in 1984, when the President's Commission on 
Excellence in Education ushered in the modem era of public school reform in the 
United States with its disquieting report, "A Nation at Risk," that Oregon turned 
out to be already working on the problem. 

Certainly, the theme of "A Nation at Risk" — that students in this country 
were being inadequately prepared to compete in a global marketplace which 
increasingly demanded the ability to solve problems, think creatively, 
communicate clearly and adapt to change— resonated in the Oregon of the 1980s. 
At the time, the State was mired in a stubborn regional recession. The long- 
standing backbone of the economy —the forest products industry —was in decline 
as a source of family wage jobs. Whatever shape the future might take, it was 
apparent to leaders that the State's youth needed to be educated to a much higher 
level of proficiency than had previously been the case. 

How large a role state government should play in fostering a 
transformation of the public schools, however, was by no means clear. Oregon 
has traditionally been a strong "local control" state. That is, schools were funded 
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mainly by local property taxes and decisions concerning how they were to be run 
were made mainly by locally elected school boards. This arrangement worked 
satisfactorily for many decades. Oregon's college-bound students scored in the 
upper third of state cohorts on national college entrance exams. Per pupil 
expenditures consistently exceeded the national average despite a state income 
slightly below the national average, a level of taxpayer "effort" demonstrating 
strong popular support for the public education ideal in Oregon. 

The forces of change were gaining momentum in the 1970s, however, 
moving along two parallel tracks: concern about school quality and school finance. 
Prior to "A Nation at Risk," the Oregon Department of Education had recognized 
the importance of no longer judging school quality by simply measuring the 
"inputs" to the system— How much instructional seat time were children 
receiving? How many books were stocked in the school library? How large was 
the average class size? ODE began to focus on results: what did students know 
and what were they able to do as a result of their schooling? In 1984, in response 
to "A Nation at Risk," the State Board of Education took the next step and 
adopted the Oregon Action Plan for Excellence. The Plan, as it was fleshed out in 
legislation in subsequent years (The School Improvement and Development Act, 
1987; The 21 st Century Schools Act, 1989) struck a mutually acceptable "deal" with 
local control advocates. On one hand, the State would lead the effort to establish 
consensus over what specific goals students should be able to reach in the 
respective academic disciplines and it would develop a statewide assessment 
system to measure the performance of all students against those standards. On 
the other, state regulators would then "get out of the way" of districts and schools, 
waiving many non-essential requirements and freeing local educators to design 
and carry out school improvement strategies responsive to their unique 
circumstances. 
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Finance Reform 



Meanwhile pressure for change was also building on the school finance 
front. As was happening across the nation, the average cost of educating a student 
in Oregon had been growing in real terms since the mid-1960s. But, with the cost 
of K-12 education underwritten mainly by the local property tax in Oregon, the 
capacity to support that increasing burden varied significantly from community to 
community. With the economy flagging, particularly in rural areas, school bond 
levies began to fail with distressing frequency. Even where levies passed, the 
amount of money spent per pupil tended to differ substantially from district to 
district, opening the financing system to charges of structural unfairness. Various 
interest groups addressed the constellation of school funding problems in the 
1980s through the popular vote, alternately trying to shift the burden through a 
sales tax (which Oregonians soundly defeated) or put a cap on the property tax 
(which they also defeated several times, but by progressively narrower margins). 

Finally, within a few months of each other, the two forces for change 
converged with decisive impact at the beginning of this decade. In November 
1990, Oregon voters passed local property tax limitation. Ballot Measures 5 (and 
its successor. Ballot Measure 47) effectively reversed the traditional relationship 
between local and state funding of schools. The local property tax share of school 
funding shrank from 60 percent in 1990 prior to Ballot Measure 5 to 29 percent in 
1995-96; meanwhile, the State share increased from 27 percent in 1989-91 to 66 
percent in 1995-97. Undeterred by the financial constraints this reversal 
portended, the Legislative Assembly overwhelmingly approved HB 3565, the 
Oregon Educational Act for the 21 st Century, in June 1991. The Act moved Oregon 
to the forefront of a national standards-based reform movement whose intention 
was to dramatically improve the academic performance of students. The major 
features of the Oregon bill— high academic goals, explicit content standards, a 
statewide assessment system to measure student performance, school site councils 
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at every school, certificates of initial and advanced mastery, and a school-to-work 
program— confirmed and extended the essential accommodation with Oregon's 
local control tradition established in the 1980s. The State would focus attention on 
the ends of education; local districts and schools would determine the means for 
getting there. 

Significantly, the State's main agent for leading the reform agenda was to 
be the Oregon Department of Education. As the law put it: 

The Department of Education shall be responsible for 
coordinating research, planning and public discussion so that 
activities necessary to the implementation of this chapter 
(Oregon Educational Act for the 21 st Century) can be 
achieved. Actions by the department to fulfill this 
responsibility and to increase student achievement may 
include. .. (etc.) (ORS 329.075(2)). 

However, although given enormous new responsibilities, ODE was not 
strengthened in size or overall budget to carry them out. In fact, separating out 
additions due to the absorption of the juvenile corrections education program, 
ODE has actually decreased in size between 1990 and the present, going from 237 
full time employees (FIE) in 1990 to 210 FTE today, a reduction in force of 
approximately 12 percent. 

THE COMPLEXITIES OF EDUCATION REFORM 

As alluded to at the outset, Oregon's reform legislation sets an ambitious 
agenda for ODE. The stated goal of the Oregon Educational Act for the 21 st 
Century is that the state's public schools are to produce "the best educated citizens 
in the nation by the year 2000 and a work force equal to any in the world by the 
year 2010." Are such lofty goals realistically attainable? So far, efforts at reform 
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have not translated into readily discernible improvements in student achievement. 
The scores of Oregon students on college entrance exams historically have 
exceeded the national average. In 1995, results on the Scholastic Assessment Test 
spiked upward another twenty points, a gain that was almost maintained in 1996. 
Nevertheless, the picture that emerges from student performance over time on 
statewide reading, writing and mathematics tests at the third, fifth, eighth and 
tenth grades is more one of stability than dramatic improvement. In mathematics, 
for example, results since 1992 have been flat at every grade level. At the present 
level of performance only one in four students will satisfy proposed state 
mathematics standards for earning the Certificate of Initial Mastery by 10 th grade. 

Does this mean ODE is failing in its reform assignment? Not yet, it does 
not. In terms of the effort, investment and time it will reasonably take to move the 
entire public school system forward, the reform movement in Oregon is still in its 
infancy. Stagnant test results certainly justify a closer examination of the 
strategies ODE is pursuing to influence instructional practice at the local level. On 
the other hand, they also suggest that the time may have come for adjusting 
collective expectations concerning how fast systemic improvement is likely to take 
hold. It is not unusual for test scores actually to drop in the early reform stages. 
"Rat" scores, therefore, are not yet cause for alarm. 

Change is possible. However, reinventing an entire state education system 
to achieve higher student performance is an enormously complex task. For 
example, improving student performance implies the need to improve the quality 
of instruction which, in turn, implies the need for professional development. But 
consider the logistics. There are approximately 25,000 teachers in Oregon. 

Granted that each subject area has a unique domain of state-of-the-art pedagogical 
approaches for teachers to acquire, that teachers often operate across the subject 
area domains, and that continuous improvement itself is messy and time- 
consuming, the possibilities for various forms of staff development in the 
curricular areas alone are extensive. Meanwhile, staff development must also be 




37 



30 



affordable, targeted where it will yield the highest payoff, and executed in a way 
that does not interfere with the main mission of schools: keeping the classroom 
alive with learning. 

The point is, setting higher standards and assessing student performance 
will not bring about achievement of the performance standards by itself. Most 
local school districts and teachers are already working as hard as they can to 
educate the State's children. To improve outcomes, teachers and schools must 
begin to work in a new way. But many educators do not know what that new 
way might be. Unless the State provides leadership in promoting appropriate 
curricular and instructional innovation and providing adequate funding and 
infrastructure support, a high percentage of students will continue to fall below 
the demanding academic standards set by the reform law. 

Effective state system reform is even more complex than already described, 
however, because it also involves extensive change beyond the school-yard walls. 
Higher education programs need to prepare new teachers to use instructional 
strategies that have proved effective in improving student learning. Higher 
education also needs to tie college admissions standards to the new performance 
standards expected of high school graduates. Businesses need to offer work- 
related educational experiences and require proof of academic accomplishment as 
a condition of employment. Perhaps the most searching changes are those 
required of the families of Oregon schoolchildren. Parents need to become an 
integral part of the education process by participating in their children's learning 
at school and at home. They need to create an environment at home that supports 
learning by reserving time to help with homework, for recreational reading and 
meaningful conversation with their children. 

In sum, school change is slow, difficult, incremental, and even generational, 
precisely because it involves social change. The good news is, Oregon's school 
reform legislation recognized this latter fact from the outset. Accordingly, ODE's 
approach to reform has been reasonably comprehensive; has recruited the 
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participation of many of the major players in the education effort; and has 
demonstrated an impressive degree of planning among its various parts. 

It is important to remember, however, that planning for change is easy 
compared to making change actually happen. Nor will ODE's faithful carrying 
out of the letter of the law, by itself, produce the desired results. Only planning 
supplemented by a comprehensive implementation strategy that offers new 
options and support to local practitioners stands a fighting chance of success. Such 
a strategy builds on a clear vision of desired student outcomes and the 
characteristics of schools capable of producing them. It grows out of a careful 
analysis of the resources that will be required to produce the desired outcomes 
compared with the resources available to do the job. And it results in a set of 
priorities for deciding what is to be done, when, and by whom. In short, carrying 
out complex reform requires that the agency charged with primary responsibility 
for its success have a detailed but flexible strategic implementation plan, tailored 
to the state's specific assets, context and policy environment. It requires that ODE 
"invent" new ways to behave as it moves further down the reform 
implementation road. This is, indeed, uncharted territory. 
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A NEW ROLE FOR STATE DEPARTMENTS OF EDUCATION 



Oregon statutes reflect two distinct functions for ODE. One is to serve as 
the principal administrative agency for the State Board of Education: 

...the department shall exercise all administrative functions of 
the state relating to supervision, management and control of 
schools and community colleges not conferred by law on 
some other agency (ORS 326.111(3)). 

The second function, cited early in this Chapter, is to lead the implementation of 
the Oregon Educational Act for the 21 st Century. 

The first of these functions is primarily administrative and regulatory and 
is the traditional oversight and fund-dispensing role that most state departments 
of education were created to discharge. The second is a much newer, more open- 
ended role requiring ODE to exercise leadership in a variety of unfamiliar ways to 
implement comprehensive, standards-based reform. So it is that the present is, 
indeed, a period of transition for ODE , with natural tensions arising as it attempts 
to reconcile its traditional identity as an enforcer of state and federal rules with its 
emerging role as a catalyst for change. 

The Oregon Department of Education is not alone in this uncomfortable 
spot. Standards-based education reform, such as Oregon has underway, is 
currently the school change strategy of choice across much of the United States. 
Enough experience has been gained with the new leadership role for SDEs that a 
few useful generalizations can begin to be made about it. In particular, leadership 
of standards-based reform seems to involve SDEs in the following five kinds of 
critical activities: 

• Setting standards and assessing student progress toward them; 

• Strengthening the capacity of local schools and districts to improve 
instruction and student learning; 
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• Conducting research and analysis needed by policy makers to guide and 
support school improvement efforts; 

• Communicating with the various publics so that all parties remain active 
participants in the reform process; and, 

• Providing essential monitoring of state and federal education programs. 

Each of these activities will be briefly discussed in turn. 

Standards and Assessment— The setting of standards and development of 
statewide assessments that accurately monitor student progress toward them is 
the starting point for standards-based reform and a basic leadership function of 
modem SDEs. Writing standards is a difficult task in itself, requiring considerable 
subject-by-subject expertise. To be effective, however, the process of standard- 
setting must build consensus and buy-in among critical constituents, including 
professional educators, parents, business representatives, and the public. And 
that takes time, effort and diplomacy. Creating statewide assessments linked to 
the subject area standards that accurately measure the breadth and depth of 
student knowledge and ability to perform is a critical task in successful reform. 
Statewide assessments are among the most powerful tools at an SDE's disposal for 
influencing what is taught in local classrooms. 

Creating appropriate assessments to statewide scale is a significant 
technical challenge. Indeed, in certain subject areas it has not yet been successfully 
carried out anywhere in the nation. It is imperative that the statewide assessment 
align with the curriculum standards in any given subject area because, as the 
saying goes in education (and research abundantly affirms) "what you test is what 
you get." Assessment is a sword that cuts two ways: good assessments will send 
the right signals to the field; but poorly designed assessments will actively 
discourage good instruction and run the risk of trivializing student learning. 
Statewide assessment results, properly disaggregated and analyzed, provide a rich 
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lode of data identifying strengths and weaknesses in the performance of the 
school system. The information, in turn, can help guide strategic decisions 
concerning the use of discretionary resources. 

Capacity Building— Another critical, on-going responsibility for an SDE in 
the modem era of reform is to assist schools and school districts to develop the 
capacity to deliver high performance instruction. This rarely involves the direct 
provision of services; SDEs are too small to make an impact on statewide school 
systems by frontal assault. Rather, capacity building typically involves forming 
alliances of service providers and making sure, through the strategic 
implementation plan, that the highest priority needs with the greatest potential 
payoff in terms of improved student learning are being addressed. A focus on 
capacity building means a major change in outlook for organizations long 
accustomed to monitoring for compliance with state statutes and regulations. In 
order to build local capacity through systematic professional development the 
modem SDE must have, for example, subject area specialists in the major 
academic disciplines who have a working knowledge of "best practice" 
instructional approaches, responsibility for organizing professional development 
opportunities, and an almost evangelical commitment to improving student 
learning in their fields. To use a sports analogy, the role of SDE personnel must 
shift from functioning predominantly as referees who dispassionately oversee the 
action on the school "playing field" to being coaches who proactively assist their 
team improve its performance. 

Research and Analysis — An organization that is focused on assistance and 
coordination of effort needs high quality research and information to guide its 
initiatives. State legislators and policy-makers also require accurate and easily 
accessible information concerning the allocation and relative effectiveness of 
resources dedicated to schools, the progress of students toward statewide 
standards, and the potential for providing educational services through 
alternative methods. Much attention should be directed to analyzing subtle trends 



implicit in the statewide assessments and using that information to direct the 
strategic implementation plan. To provide timely and accurate information to 
policy makers, modern SDEs also need an operational management information 
system and financial information system. 

Communication — School improvement is not something than can be 
accomplished by an SDE acting alone. It requires the support and involvement of 
a host of coequal participants: teachers, administrators, the curriculum 
associations, school boards, parents, business leaders, higher education programs, 
and, of course, students. SDEs must do an effective job of communicating with 
these various "publics" about the goals and strategies for school improvement. 
This communication must be more than mere reportage. A consistent message, 
which reinforces the State's overall education improvement strategy, must be 
crafted and tailored for various audiences. 

Essential Monitoring — There will always be an irreducible set of federal 
and state requirements that SDEs must monitor. Rules to protect the health, safety 
and civil rights of children must be closely observed. Federal and state statutes 
must be obeyed, among other reasons, to preserve a state's fair share of financial 
support for categorical programs. Additionally, in this era of contemporary 
education reform, the monitoring and enforcement function focuses much more 
intensely on monitoring for results - what students demonstrate they know and 
are able to do - and, implementing a system to hold districts and schools 
accountable for students' achievements. 

Taken together, the five functions described above define the leadership 
role of a modem SDE. Leadership does not mean providing a technical "fix" for 
every problem that comes up. Leadership for reform means building a broad- 
based consensus around a coherent vision of improved teaching and learning. 

That vision must then be supported by concrete strategies for achieving it over the 
long term. Effective SDEs become the facilitator of broad coalitions of public and 
private partners who are committed to the achievement of high academic 
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standards. The hard work of improving schools will be done in communities and 
in schools. SDEs support and lead this work— through standard setting, 
continuing assessment, capacity building, research and analysis, communication 
and essential monitoring. 

The citizens of Oregon have undertaken a major challenge: namely, 
substantially to improve the academic performance of students in the State's 
public schools. At the same time, Oregon has moved to a state-based system of 
education finance. The Oregon Department of Education has a critical leadership 
role to play in the education reform effort. The next Chapter of this report details 
MAP's findings concerning how ODE has performed in carrying out this 
important work. 
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Chapter 3 



FINDINGS 



This chapter displays the findings from this evaluation study. These are the 
data-driven conclusions at which MAP arrived after reviewing, analyzing, and 
synthesizing all of the collected information on the Oregon Department of 
Education. The findings are divided into five categories: 1) leadership, 

2) monitoring and enforcement, 3) capacity building, 4) research and analysis, and 
5) communication. 

Prior to offering the findings, MAP wishes to make two points which we 
believe readers must bear in mind. The first is that we were impressed with just 
how hard ODE employees work and how dedicated they are to doing their jobs 
well. Often in evaluations of state agencies, we encounter employees who are 
"just getting by," putting in the hours, but not committed to the work. This is not 
the case at the Oregon Department of Education. Staff members, regardless of 
their title or job description, are working hard to accomplish very difficult tasks. 
That leads to the second point. 

What Oregon is attempting to do— implement a standards-based system of 
education— is enormously complex. This point cannot be stressed too forcefully 
or too often. No state has yet completely implemented the kinds of education 
reforms underway in Oregon. Policy makers, citizens, educators, and ODE 
employees themselves should not underestimate just how hard the work is or how 
long it will take to "get it right." 

A final reader's caveat: Some of the findings in this chapter may sound a bit 
harsh. They are not meant to be that at all. Findings are offered in the spirit of 
assisting ODE successfully to accomplish the work before it. We hope readers— 
and ODE staff— will take them in the spirit in which they are offered. 

We turn now to the first category of findings— leadership. 



LEADERSHIP 



Leadership is often a rather ineffable quality, sometimes difficult precisely 
to define. Yet we all know leadership when we see it. In other words, the product 
of leadership frequently is more apparent than its performance. 

For purposes of this study, MAP has defined leadership as articulating and 
promoting a common vision of education, a common set of goals and expectations, 
and means by which those expectations can be met. Neither prescriptive 
legislation nor sophisticated management procedures and techniques can 
sufficiently compensate for inadequate leadership. 

Leadership for education needs to be exercised at multiple levels and by 
many offices, organizations, and individuals. The governor has an opportunity to 
display leadership through public pronouncements (the often underestimated 
"bully pulpit") and budget priorities. Legislative leadership comes through in the 
kinds of statutes approved (the "what" of education improvement) and the 
commitment to a sometimes rocky, but ultimately productive, course of action. 

The state board of education and various interested groups of citizens and 
professional educators have opportunities for important leadership roles as well. 

Then, of course, there is the leadership it is incumbent on ODE to exercise. 
In this particular period, when education improvement is so high on the public, 
and often the political, agenda, ODE must be able to exert leadership through 
influence and example, incentive and exhortation, rewards and consequences. 

ODE must put forth a clear and consistent message about what education 
improvement means, how it will be accomplished, and the consequences for 
failing to make forward progress. The Oregon Department of Education must, in 
short, be the continual catalyst for improving teaching and learning. 



FINDING # 1: 



ODE employees, while dedicated and hardworking, do not perceive 
themselves as having an important role in improving teaching and 
learning. 

In interviews and visits to various ODE offices, MAP was 
continually struck by how hard people are working. There is a level of 
dedication to the job that seems able to overcome whatever natural 
frustrations go hand-in-hand with trying to chart new territory. 

This ethic of hard work and determination is communicated by ODE 
staff to educators in ESDs and local school districts. On the survey 
conducted for this study, when administrators were asked to name positive 
qualities of ODE, "good people, personable, helpful, dedicated" and 
"competent and professional" staff were the most frequently volunteered 
responses. 

However, process— understanding laws, ensuring compliance with 
the letter of the law, planning for various kinds of activities, and 
developing new programs, for example— too often takes precedence over 
results. To illustrate: MAP presented top ODE managers with a 
hypothetical scenario. Suppose that there was a significant decline in 
reading scores statewide. What, MAP asked, should ODE do? 

Virtually all of the ODE staff with whom we discussed this agreed 
that it was ODE's responsibility to report the decline in test scores. But, 
they then asserted, the responsibility for decline in performance— and the 
responsibility for improvement— would rest solely on the shoulders of local 
educators. ODE, they said, could not be held accountable for student 
performance. One high level ODE manager told MAP, "It is not our job to 
make sure that scores are going up." Another declared, "The department 
[ODE] has nothing to do if test scores are not up to standard. It would 
require legislation [for ODE to take action]." 
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This last statement is perhaps the boldest example of an ODE 
misconception. On the one hand, the Oregon Department of Education has 
the responsibility to implement the reform law enacted by the legislature. 
On the other, ODE has so narrowly construed and so literally interpreted 
the language of the statute that the agency has actually handcuffed itself. 

ODE seems unable to see beyond a strict legal, point-by-point 
interpretation of a very complex and encompassing law. The agency seems 
not to understand that within the rather broad parameters of the statute, 
ODE must articulate the vision for reform, and initiate a coherent set of 
strategies designed to bring it to fruition. 

When MAP would question ODE officials about what they were about 
and why, the most frequently offered response was, "because it's in the 
law." In short, ODE seems not to have grasped that the reform statute 
provides only the basic blue line drawings for education improvement. 
ODE must transform those drawings into an actual structure. 

In probing a little more deeply and discussing specific strategies 
ODE might employ to focus on improving teaching and learning, MAP 
again turned to the issue of testing. Assessing student achievement is, in 
MAP's judgment, one of the most powerful tools ODE has at its disposal. 

ODE managers were asked which best describes statewide student 
assessments: Are assessments a thermometer or a lever? Responses were 
consistent: assessments are a thermometer. They are, in other words, say 
ODE officials, a means of reporting a score and accomplishing a legislative 
mandate. Assessments are not seen as a tool for education improvement, 
an indicator of where and how ODE might target personnel and resources 
to ensure that where scores are high (or climbing), districts and schools 
have encouragement and incentives to keep up the good work, and where 
they are low, districts and schools have support and assistance to bring 
about improvement. 
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This is a particularly important point. Assessments are not merely an 
analytical tool. They are a key means by which to send a message about the 
kinds of learning standards that are expected and required. 

MAP also found that almost none of the ODE managers and 
employees interviewed was able to articulate a coherent vision about how 
schools in Oregon should change to meet the state-established student 
achievement standards. Among those who offered an opinion there was no 
obvious agreement. ODE staff jobs were defined by laws and there was no 
responsibility for interpretation or room for initiative. 

MAP believes that it is ODE's legal compliance view of education 
reform that is holding back additional progress ODE— and schools and 
districts — might be making. MAP certainly does not advocate that ODE 
violate the law. Rather, we are concerned that in its devotion to the letter of 
the law, the spirit of the statute— and the best opportunities for improving 
education— may be out of ODE's line of sight. 

The imperative for ODE staff to devote their time and attention to 
matters of teaching and learning is growing ever stronger. We are 
convinced that, as matters now stand, large numbers of Oregon students 
will not achieve the Certificate of Initial Mastery (CIM) on schedule. Yet 
ODE seems to have no contingency plan for this eventuality. Here is an 
instance in which ODE leadership is essential if the State wants to ensure 
that education reform is not entirely derailed. 
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FINDING # 2: 



ODE's current organizational structure neither conveys nor facilitates an 
activist leadership role in improving teaching and learning. 

Currently, two deputy superintendents and seven assistant or associate 
superintendents report directly to the superintendent. Only one associate 
superintendent reports through a deputy. The organizational units 
reporting to the superintendent are Special Education; Professional 
Technical Education; Student Services; Compensatory Education; 
Curriculum, Instruction, and Field Services; Assessment; State Board 
Relations; and Education Support Services. Also listed on the ODE 
organization chart are Drug and Alcohol Education, Teen Parent Program, 
Homeless Program, and Diversity Program. Conspicuous by their absence 
are curriculum areas such as reading, mathematics, history, science, and 
English. 

The organization of ODE reflects a preoccupation with programs and 
activities, many of which are either funded through categorical grants (and 
often in departments of education set up as nearly freestanding entities) or 
are generic in nature (Student Services, for example). We do not mean to 
suggest that these programs and activities do not have educational value; 
all probably do. The problem is that the organizational structure does not 
telegraph a clear departmental mission, a focus on the essentials of 
education— how teachers teach and what and how students learn. 
Assessment is given equal organizational weight as special education; the 
specific subject areas that form the core of the assessment system are 
nowhere to be found. 

We want to give some special attention here to the issue of subject 
matter experts. In our judgment, experts in the fields that are the central 
focus of the State's assessment efforts are an essential component of ODE 



staff. We do not mean to suggest here that these individuals should alone 
occupy the ground in a particular field. Quite to the contrary, ODE subject 
matter experts should be the liaisons to other organizations, such as 
curriculum associations, and promote and coordinate professional 
development in their particular field. 

ODE currently has only a part-time science expert on staff and a 
mathematics position which has remained unfilled since June of 1996. MAP 
is aware that some ODE officials believe that they were required by the 
1995 amendments to the Educational Act for the 21 st Century to eliminate 
subject matter specialists from ODE positions. Yet the 1995 amendments, 
properly, do not speak to the organization of ODE. 

What did happen was that ODE was required (again) to reduce its 
budget, and made a decision to reassign its eight subject matter specialists 
to more generic positions, and then not to replace those specialists who 
retired. ODE found itself between the proverbial rock and hard place. It 
had been experiencing progressively reduced resources (Curriculum, 
Instruction, and Field Services, for example, has had its staffing cut by 
nearly 40 percent since 1990) at the same time as it was being called upon to 
carry out a whole new, and quite complex, set of responsibilities. The 
choice ODE made was to abandon subject matter expertise. As will be clear 
in the recommendations section of this report, MAP believes this is a choice 
that must be reconsidered. 

Creating a coherent vision for reform and implementing a common and 
coherent strategy to achieve it requires constant communication by ODE 
staff within and across organizational units. Yet interviews with ODE 
managers and employees reveal a consistent theme of insufficient 
communication among and between program units. Interview results 
suggest a strong tendency within ODE to compartmentalize work. 



Specialists in the Assessment Office, for example, describe their 
communication with ODE's few curriculum specialists as episodic and 
difficult, in large measure because the curriculum specialists have 
responsibilities beyond curriculum leadership and spend considerable time 
out of the office. Even in instances in which management reports unit 
collaboration, follow-up interviews revealed that this was more 
serendipitous than systematic, sometimes more desired than real. The head 
of one office, for example, described close collaboration between specialists 
in that office and staff of another office. Follow-up interviews with 
specialists in both offices revealed that, while there was a strong interest in 
collaboration, the workload of each office precluded it. 

FINDING # 3: 

Current conditions preclude ODE from securing as employees the broad 
range of educational experts necessary to carry out the requirements of 
education reform. 

The challenge to attract leaders and experts to ODE roles is 
compounded by the fact that salaries for education professionals at ODE 
lag significantly behind those in Oregon school districts. A comparison of 
ODE and school district salaries is displayed below: 

Average salaries in Oregon districts with 3000+ ADM: 



Position Salary 



Deputy and Area Superintendents $88,423 

Assistant Superintendents $76,828 

High School Principals $70,390 

Elementary Principals $62,305 

Teacher with MA+45 units $47,733* 



*median of 190 days per year 



Source: OSBA Salary Survey 
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Top of salary schedule for ODE ( year-round employment): 



Position 


Salarv 


Deputy Superintendent 


$75,132 


Assistant Superintendent 


$62,784 


Specialists 


$52,956 



Source: OSBA Salary Survey 

These differences are even more dramatic when compared to Oregon's 
largest districts, those with more than 10,000 ADM, where salaries are even 
higher. 

The last general salary increase for ODE employees occurred in 1993. 
Since then, salaries have been static, and the gap between district and 
school salaries and those for ODE employees has grown. 

As a stark example that ODE is not competitive in the job market, it is 
interesting to note that local school districts often hire good people away 
from ODE, not vice versa as is usually the case. Moreover, given the range 
across the state in terms of teachers' salaries (a teacher at the top of the 
schedule in Bend earns $42,125, in Portland $46,081, and in Beaverton 
$52,728), ODE will be most able to recruit inexperienced teachers from rural 
areas who, while dedicated and committed, are unlikely to have worked in 
large organizations or with complex institutional change. 

This situation regarding ODE's competitiveness in the job market is 
particularly acute in positions requiring technical expertise. ODE's highly 
regarded assessment manager recently left for a higher paying job. As 
previously indicated, ODE has been unable to fill a mathematics position 
since June 8, 1996. 
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"Low salaries" of ODE employees is cited as a problem in interviews of 
school district, ESD, and higher education representatives. ODE personnel 
officers and managers report a significant decline in the number of 
candidates for advertised ODE positions and a perceived "lower quality" of 
the candidate pool. Fewer candidates, they say, possess the training and 
experience the jobs require. Many lack direct K-12 experience, and among 
those who do have K-12 experience, many come to ODE directly from the 
classroom, with little managerial or leadership experience that would give 
them the broader perspective needed for their statewide roles. ODE 
personnel officials also report that fewer job seekers are from urban and 
suburban districts. 

It may reasonably be claimed that ODE would pay more for staff if it 
had more dollars available. Thus, in an effort to determine the State's level 
of effort, as measured by dollars spent toward ODE, MAP compared 
Oregon with six states of comparable size. The comparison states— three 
somewhat smaller than Oregon and three somewhat larger— were Kansas, 
New Mexico, South Carolina, Kentucky, Connecticut, and West Virginia. 
The average enrollment in these states is 484,840 students, as compared to 
Oregon's 502,711. Results of this comparison reveal that the average cost 
per pupil associated with operating the state education agency in the 
comparison states is $49.63. In Oregon, the average cost per pupil is $41.15, 
slightly more than $8 per pupil below the average of the comparison states. 

If ODE were funded at the average of this group, its annual budget would 
be approximately $4 million larger. 

While these figures do not necessarily mean that ODE is underfunded 
(at least not seriously, relative to other state education agencies), it is also 
clear that ODE is not over-funded. Moreover, the heavy responsibilities of 
ODE to implement a very complex set of education reforms must also be 
considered part of the funding mix. 
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MAP is aware that ODE salaries are subject to state laws and policies of 
the Department of Administrative Services and, for non-managerial 
employees, collective bargaining. Our review of applicable statutes reveals 
a strong interest in maximizing diversity in the workforce and minimizing 
job classifications and salary differentials for equally valued work. (See, for 
example, ORS 240.190, 240.215, 292.971, and state policy 20.000.01). ODE 
specialists are part of the same bargaining unit as most other state workers. 
Because they are a very small part of the bargaining unit, their interests 
rarely are specifically represented in negotiations. Problems 
notwithstanding, there are procedures available that could address gross 
disparities between ODE salaries and those in school districts, but they 
have not been employed. 



FINDING # 4: 

ODE does not systematically employ comprehensive planning and 
budgeting as management tools. 

Based on interviews with ODE managers and employees, MAP 
concludes that planning at ODE is episodic and ad hoc. To be sure, priority 
activities are planned. However, priorities are something of a moving 
target, and tend to proliferate. This may be the natural consequence of a 
rapidly shifting social and political environment. What seems to be lacking, 
however, is an annual "baseline" from which reasoned deviations might be 
made. 

In addition, ODE managers reported to MAP that they have little 
responsibility for managing their own budgets and, in fact, most display 
little knowledge of, for example, the remaining balance of their budgets for 
which they are, at least ostensibly, responsible. Managers assert that there 
are no consequences for overspending on their budgets, so this is a matter 
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to which they pay little attention. The Superintendent confirmed that 
nearly all budget decisions are made by her office. 

FINDING # 5: 

Few statutory changes are required for ODE more effectively to take a 
leadership role in improving teaching and learning in the state. 

Most of the changes MAP will recommend to enhance ODE's 
effectiveness on the dimensions of improving teaching and learning can be 
accomplished through altered management or administrative structures. 
This is not a case in which serious legislative intervention is either 
necessary or warranted. 

In fact, MAP is loath to tamper much with statute. In our judgment and 
based on our experience, laws should be lean and generally not 
prescriptive. Detailing processes and procedures in law leads inevitably to 
disappointing results. Laws cannot anticipate every eventuality, 
encompass every circumstance, or make provision for every occurrence. 
Laws governing an agency, such as ODE, should provide a basic set of 
operating guidelines, policy maker expectations, and citizen safeguards. 

MAP's criterion in assessing whether or not sections of education code 
should be added or subtracted was a simple one: Does a law interfere with 
ODE's primary mission of improving teaching and learning or, 
alternatively, is a statute required to enhance ODE's opportunity to 
accomplish its primary mission? 

On the latter count, whether new laws are needed, we come down 
firmly on the "no" side. Current statutes adequately offer ODE 
opportunity and flexibility to provide leadership to improve teaching and 
learning in Oregon districts and schools. 
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We make special reference here to two existing sections of law which we 
believe should be changed. The first is the requirement that ODE conduct 
school improvement visits to every district in the State on a staggered 
three-year timetable (ORS 329.085). In our judgment, these visits divert too 
much ODE staff time from necessary functions and have too little payoff for 
them to be continued in their present form. (See the chapter section on 
Monitoring and Enforcement for a more complete description of this issue.) 
We would recommend that the statute on school improvement visits be 
eliminated, or at least substantially modified. 

Second, we believe that the law which requires ODE to administer the 
criminal background checks for all noncertified school employees (ORS 
326.603) (actually, to do the requisite paperwork; the check is conducted by 
the state police) places an unnecessary burden on ODE. As detailed later in 
this report, we believe that this responsibility can be removed from ODE's 
jurisdiction, thus freeing ODE staff for other duties. Finally, as described in 
the next finding and in the recommendations section of this report, MAP 
believes that the current situation of an elected state superintendent is 
problematic. Changing this will require legal or electoral action. 

FINDING # 6: 

The elected office of superintendent of public instruction tends to 
fragment responsibility for education. 

The Oregon Constitution at present requires that the state 
superintendent be elected. Yet, an elected chief state school officer 
fragments responsibility for education. The superintendent is responsible 
for providing programmatic leadership, while the governor is responsible 
for the state budget. This arrangement tends to relieve both the 
superintendent and the governor from total responsibility to the citizens of 



Oregon. The superintendent can claim that schools fail to perform because 
of inadequate funding and the governor can point to inadequate 
programmatic leadership. This situation can be exacerbated when each 
office holder is a member of a different political party or when the governor 
and the superintendent hold highly divergent philosophical views about 
education . 2 

Environmental changes have added some urgency to the need to 
streamline governance arrangements and establish clearer lines of 
authority. In particular, the greater state role in financing schools creates a 
fiduciary responsibility on the part of the State to account for the way in 
which dollars are spent. The governor and the legislature currently are the 
responsible parties. The superintendent, though an elected constitutional 
officer and head of the agency responsible for implementing the State's 
complex reform statute, has quite limited budgetary authority. Dollars are 
appropriated to ODE functions, but in many instances, ODE simply acts as 
a "pass through" to local school districts. The real authority for 
determining levels of funding resides with the governor and legislature. 

A different arrangement, detailed in the Recommendations chapter of 
this report, would both provide a formal link to the policy makers who 
control state dollars, and would provide an opportunity reasonably to 
ensure for the future that the State's chief school official can both work 
effectively with the State's chief executive officer and has the requisite 
background, expertise, and training to carry out a critical State function. 

MAP points out here that the discussion of an appointed superintendent 
falls in the "good government" realm. There is currently no crisis in 
Oregon on this dimension. However, diffuse responsibility for education, 

2 Neither of these conditions seems to prevail in Oregon at present, but both are possible under the current 
arrangement 
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as is now the case, seems inconsistent with the State's taste for 
accountability. 
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MONITORING AND ENFORCEMENT 



Monitoring and enforcement conventionally implies ensuring that some 
set of laws, rules, and regulations is adhered to appropriately. For state 
departments of education, this task historically meant making sure that 
districts and schools hewed faithfully to the various statutes promulgated 
by the state and federal branches of government. For modem state 
education departments, however, in other words, for those that are 
evolving new roles more consonant with the obligations of systemic 
education reform, "monitoring and enforcement" must take on a new 
meaning. Increasingly, departments of education must translate the 
"follow the rules" sense of monitoring and enforcement to ensuring that 
districts and schools successfully implement more effective teaching and 
learning. Departments of education then monitor and enforce the extent to 
which these efforts are successful; in other words, the extent to which new 
student achievement standards are being met. 

FINDING #7: 

ODE has undertaken efforts to transform itself from a regulatory 
organization to a school assistance organization. However, 
despite its efforts to move from regulation to assistance, ODE continues to 
conduct most of its work as if it were primarily a regulatory and 
compliance organization. 

Interviews of ODE employees reveal a strong desire to move away from 
conventional monitoring and compliance activities and toward greater 
assistance to schools and districts in improving teaching and learning. The 
Superintendent and her staff are to be commended on the changes they 
have made, for example, to focus more on technical assistance to schools 
and districts. 
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Clearly ODE leadership is endeavoring to refocus the primary 
manifestation of the enforcement norm, the School Improvement Visits, 
away from compliance and toward assistance. Oregon law requires that 
every three years the State Board of Education or its "designee" (read: 

ODE) assess the effectiveness of each public school district in an on-site 
visit. These school improvement visits, as they are called, replace the 
former standardization visits. 

Previously, the visits' sole purpose was to provide on-site verification 
that the district was maintaining adequate records of compliance with state 
regulations and meeting other requirements designed to protect students' 
health and safety. Now these visits also are meant to determine whether a 
district's school improvement plan, and individual schools' school 
improvement plans, are likely to be effective in achieving state curriculum 
goals and standards. 

Respondents to the MAP survey conducted for this study clearly are 
aware of this change. They report that their site visit discussions with ODE 
officials recently have focused, to at least some extent, on standards and the 
degree to which their students are meeting them. 

However, accumulated evidence from MAP's review of documents, 
interviews with ODE administrators and school district personnel, and 
observations suggests that ODE personnel still spend too much of their 
time on regulation and compliance and too little on assisting districts and 
schools to improve teaching and learning. 

The evidence for this finding comes from several sources. Much of the 
problem stems directly from statutes and the Oregon Administrative Rules 
that direct ODE to enforce specific policies. Let us turn again to the school 
improvement visits. 

ODE is required by law to make school improvement visits to each 
school district every three years. While we found general compliance with 
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this requirement, MAP was informed by ODE managers that ODE would 
not conduct visits to the required one-third of districts in the current year 
and was seeking legislation to relax that requirement to one-quarter 
annually. A more fundamental question raised by the visits is whether this 
is an effective use of ODE's limited resources. 

School improvement visits consume, according to ODE sources, 
approximately 1250 professional days of ODE personnel each year. A 
typical visit involves on average eight ODE staff members spending two 
days visiting the district office and various schools in the district. In 
addition, the team leader and assistant leader make a previsit to review 
district documents and establish a schedule for the main visit. After the 
visit, the team prepares a report that must be given to the district within six 
months. 

The amount of time spent by school district and school personnel in 
preparing self-studies in anticipation of the school improvement visits is 
considerable as well. The majority of administrators interviewed by MAP 
did not believe this to be an efficient or effective use of their time. 

School improvement visiting teams are also to provide technical 
assistance to schools and districts, as requested. However, ODE staff who 
are available on a scheduled visit day compose the ODE teams. Staff skills 
and expertise are not necessarily matched to district or school needs. (On a 
school improvement visit attended by MAP, for example, a bilingual 
specialist whose particular expertise was not required at the school was 
dispatched to review the process for dispensing drugs in the nurse's office.) 
In the absence of ability to provide particular assistance on a given topic 
(reading improvement, for example) the teams understandably resort to 
compliance checks— how the nurses supply cabinet is stocked, whether fire 
drill directions are clearly posted, etc. 
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MAP's general observation is that school improvement visits are 
enormously time consuming for ODE personnel and the purpose of the 
visit is not well understood by district personnel or ODE visiting teams. 
Moreover, confusion still reigns regarding whether these visits are 
designed to enforce legal compliance or facilitate school improvement and 
if the former, how that is to be achieved. 

ODE is also required by law to administer the criminal background 
checks procedure for all noncertified school employees. The actual checks 
are made by the state police, but the processing of the paperwork takes the 
full-time involvement of several ODE employees. While ODE is apparently 
in compliance with the statutes requiring this activity, this is another 
example of the kind of required administrative duty that consumes an 
inordinate proportion of ODE time. It is not surprising therefore to find 
that the ODE is organized around these administrative functions. 

Personnel are assigned to monitor and enforce specific programs rather 
than to achieve major educational goals. 

Interviews with ODE administrators reveal the depth with which 
compliance issues dominate their thinking. They will talk about being 
service oriented and providing technical assistance to school districts, but 
when asked why certain things are done, they often say because the law or 
a rule requires it. There is little connection made between education goals, 
a vision of reform, and a coherent strategy to move from status quo to 
higher standards of achievement on the one hand, and activities in which 
ODE personnel are engaged on the other. 

We discovered that reports sent to ODE are often ignored or seldom 
used. For example, every school district is required to submit to ODE by 
January 1 a document entitled, "Annual Report on Compliance with 
Minimum Standards." When one division director was asked how the 
extensive amount of information in the report was used the individual 
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claimed no knowledge of the report. Much compliance work continues 
because it is required, not because it is useful. 

At the same time, most school administrators and teachers have little 
direct contact with ODE personnel. Not one school principal or teacher we 
talked with knew the name of, or had ever had contact with, ODE's official 
liaison person for schools in their region. Most administrators felt that 
ODE is too distant from actual school operations to be of much help to 
them. 

The perception that ODE is too rule-bound is reflected in the results of 
the survey conducted for this study. "Bureaucratic" is the word the 
majority of respondents used to describe ODE. "Creative" and "flexible" 
do not rank high as ODE descriptors. 

When asked what role ODE plays in Oregon, the largest group (46%) 
responded that it acts as a referee; 37 percent said it acts like a coach. This 
is in stark contrast with administrators' preference that ODE be more of a 
coach. Eighty-nine percent preferred this role, while only 7 percent 
preferred the referee or spectator role. Moreover, it is important to point 
out that those administrators who perceive ODE as performing a coaching 
role are at least twice as likely to rate the performance of ODE as excellent 
or good than are those who view ODE as more of a referee. 

When asked to say what ODE does well, respondents listed managing 
federal programs, enforcing state regulations, and assessing students. Less 
than a third of the local administrators think ODE does a good job of 
supporting staff development and program assistance, providing financial 
management assistance, and helping with local improvement efforts. In 
other words, ODE is perceived as doing a good job of carrying out its 
administrative responsibilities but a less effective job of supporting 
educational reform. 





58 



Why is this all important? Many of ODE's regulatory activities are not 
very useful (though they may currently be mandated) and consume 
resources needed for other important activities. It is not necessary to check 
that every school holds earthquake drills as required by law. That 
responsibility could be transferred to school districts, or a small sample of 
districts could be checked for compliance. Rather than assuming districts 
are not complying with rules, the assumption should be that they are. 
Procedures can then be devised to root out the few schools or districts that 
are derelict. 

The main problem with a compliance orientation, however, is that it 
takes time and resources away from the more important task of assisting 
schools improve instruction and learning. There is so much work to do 
with limited resources. Ways must be found to reallocate ODE resources to 
the improvement of education in the State. 

FINDING # 8: 

ODE has instituted a statewide assessment program but may not be sufficiently 
positioned to take full advantage of its power to improve teaching and learning. 

Given the imperative of standards-based education reform, the 
traditional monitoring and compliance function of State Departments of 
Education is shifting to an emphasis on measuring the progress of students 
toward agreed-on statewide academic goals. The Oregon State Assessment 
Program is the piece of Oregon's education reform strategy that ties higher 
academic standards to the content and performance standards defining 
what students should know and be able to do in the areas of English 
Language Arts, mathematics, science, the social studies (history, 
geography, economics, and civics), the arts, and a second language. 

Developing good assessments - tests that are fair, reliable, valid and which 
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send the right signals to the field - is a complicated, and in certain of these 
subject areas on a statewide scale, pioneering endeavor. Lack of resources 
and the increasing time pressure to "get it finished" can be a recipe for 
disaster. Oregon, so far, has avoided some of the worst pitfalls. 

ODE officials responsible for assessment have done a commendable job 
of beginning to develop a statewide assessment program. Amendments to 
the Oregon Educational Act for the 21 st Century made by the 1995 
Legislative Assembly changed the purpose of the statewide assessment to 
focus on determining if students have met the standards established for the 
Certificate of Initial Mastery at approximately grade 10. Benchmark 
assessments at grades 3, 5, and 8 are to be used to determine if students are 
making appropriate progress towards achieving the QM on schedule. 

Thus far, criterion-referenced multiple choice and/ or open-ended tests 
have been put in place at the various developmental levels in reading, 
writing, and mathematics and have been well received. For example, 
public school administrators and principals in the MAP survey gave ODE 
good marks for its performance in this area thus far. The writing test, in 
particular, has had a marked positive impact on the amount and quality of 
writing being done by Oregon students. ODE staff told us that the scoring 
rubric for writing is frequently in evidence in the classrooms they visit 
around the State. 

Statewide assessments are currently under development in science and 
social studies. Local districts are to identify the assessments and standards 
for acceptable performance in second languages and the arts. 

Requirements for Health and Physical Education are described in Oregon's 
Essential Learning Skills and Common Curriculum Goals but are not the 
subject of statewide testing. 

Much work remains, in short, and there are reasons for caution. Some 
of the subject matter associations are not confident that assessments in their 
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fields properly will measure what students should know and be able to do. 
There is a danger that if assessments are designed to be executed "on the 
cheap," they will not be appropriately aligned with the curriculum; in other 
words, they will not measure (and therefore, not encourage teachers to 
emphasize fostering) the knowledge and skills needed to carry out complex 
tasks. In science, for example, ODE has decided not to include a bench or 
lab test in the statewide assessment and may not include an open-ended 
instrument. However, the consensus of science educators is that hands-on 
learning is essential for helping students to start thinking scientifically. If 
students are not tested for competency in experimental design and concrete 
problem-solving, would it be surprising if many science courses in the state 
continue to not offer these kinds of experiences to students? 

The point is, statewide assessment is an extremely powerful tool for 
influencing what happens at the classroom level, but it is also a two-edged 
sword. Good assessments will support the goals of education reform just as 
bad assessments will undermine them. Here is a prime opportunity for 
ODE to exercise leadership - to be sure that state policy makers 
understand that adequate resources are necessary to develop tests and that 
they are apprised of the level of resources that an on-going, effective, 
statewide assessment program will require; to acquaint administrators and 
teachers with the nature of the assessments, and to provide in-depth 
professional development for teachers so that they can prepare their 
students to achieve at high levels. In addition, ODE has a continuing 
responsibility to help the public understand what these tests are, and what 
the scores mean. 

MAP found little evidence that ODE is prepared to assume these 
various roles. Of course, there will always be a tension between the quality 
of statewide assessments and the cost of administering them (extrapolated 
by ODE at around $4 million per year by 1999 when the Oregon State 
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Assessment Program is fully in place). Unfortunately, the finding already 
alluded to that ODE managers and staff are more likely to think of 
assessment as a thermometer for sampling student performance rather than 
a lever for influencing the quality of teaching and learning at the classroom 
level suggests a fundamental lack of understanding of the dynamics of 
education reform and does not inspire confidence in ODE's capacity to lead 
in this area. 
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CAPACITY BUILDING 



Implementing the complex changes implicit in the Educational Act for 
the 21 st Century requires a tremendous investment on the part of the state 
in equipping local districts and schools to improve teaching and learning. 
What is called for in Oregon's reform statute are new ways of conducting 
education's business— higher standards, new kinds of curricula, new 
teaching methods, new forms of assessment to measure progress, and, of 
course, greater expected levels of student achievement. 

Achieving the primary goal— increased student achievement— is not a 
matter of simply telling districts or schools to pay more attention or 
teachers to work harder. This is not an instance in which the "bully pulpit" 
serves education well. 

Districts and schools, and teachers in the classrooms, are working at 
capacity now. The challenge is to assist districts and schools to organize 
differently, behave differently, and then set their sights even higher. Their 
efforts must be geared around the State's expressed vision for education, 
and they must be offered adequate support and assistance so they can meet 
the expectations established for them. What we are talking about here is 
what classically is called professional development. Professional 
development often has a bad reputation among teachers— and deservedly 
so. National studies have shown that offerings tend to be only loosely 
related, if at all, to teachers' expressed needs or professional 
responsibilities. Staff development tends to consist of structured courses 
which frequently are offered on "tired time" after teachers have taught all 
day. Programs are organized in one-shot sessions, and financial investment 
is generally low. 

Literature on professional development suggests that effective efforts: 
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• stimulate and support site-based initiatives; in other words, are 
closely linked to school efforts to improve teaching practice; 

• support district, school, and teacher initiatives; 

• are structured around the knowledge base of teaching; 

• offers opportunities for teachers to broaden and deepen their subject 
matter knowledge; 

• offer teachers opportunities to be active learners; in other words to 
explore, question, and debate new ideas in order to integrate them into 

their teaching repertoires and classroom practice; 

• provide adequate time and follow-up support for teachers to master 
new strategies and content and integrate them into their practice; and, 

• are accessible and inclusive, available to all as a valued part of 
teachers' work. (Corcoran, 1994). 

The goal of this sort of ongoing professional development is to build the 
capacity of teachers, and of schools and districts, to be self-renewing 
learning organizations that constantly seek out new knowledge and 
appraise and revise professional practice based on research and experience. 

ODE has an essential role to play in this capacity building endeavor as a 
partner with districts, schools, and teachers. To be sure, there are other 
forms that state-offered technical assistance might take (for example, 
assisting districts with financial management matters and removing 
unnecessary regulatory or bureaucratic impediments), but it is in this 
capacity building arena that the biggest payoff is possible. 



FINDING # 9: 



Many key ODE staff members convey an understanding of the 
importance of providing ongoing assistance to districts and schools. 
However, ODE has no apparent comprehensive plan for building local 
school and district capacity through ODE-proffered technical assistance. 

From interviews with ODE staff members, reviews of ODE documents, 
and reports of speeches by ODE officials, it is clear that key ODE staff 
understand how critical reform-related "technical assistance" for districts 
and schools is to Oregon's education improvement effort. Developing a 
consciousness about the need for ongoing professional development is a 
critical component of assisting districts and schools to build capacity for 
change. 

The Superintendent, in her address to the State Board of Education 
outlining ODE's major goals for 1995-97 stated: "Staff development must 
become a systematic and collaborative effort with school boards, 
administrators, and teachers. We need to work with teachers and 
cooperatively plan to use staff training time in a focused way to bring about 
school improvement." 

Oregon's Goals 2000 plan, written by ODE to secure federal dollars, 
refers several times to "bottom up" reform, noting that, "change must 
emerge from the school site ... and should be based on professional 
knowledge and a solid foundation of research." 

ODE's 1995-97 Legislatively Adopted Budget document declares, 
"Department staff will provide necessary technical assistance and staff 
development ybr schools so that they may reach individual school improvement 
goals" (emphasis added), and, "The Department of Education will become a 
clearinghouse for dissemination of research findings through an interactive 
network to communicate with schools...." 
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These are illustrative examples which demonstrate that ODE is aware of 
the need for assistance to schools and districts, can speak knowledgeably 
about the goal of technical assistance efforts ("so that schools may reach 
individual school improvement goals"), and understands that ODE cannot, 
on its own, offer the kind of sustained professional development required 
for education reform (ODE "... will become a clearinghouse"). 

However, here is a clear case in which actions really do speak louder 
than words. The "actions" — the kinds of assistance provided by ODE to 
districts and schools— often belie both written descriptions and public 
pronouncements by ODE officials. Interviews with ODE staff confirm that 
"technical assistance" consists primarily of providing information about 
rules and administrative requirements (for example, how to complete an 
application for funding). Moreover, several ODE managers and specialists 
reported to MAP that, increasingly, this function is being performed by 
support staff as professional staff members' time is consumed by, for 
example, school improvement visits. 

When asked about a strategic plan for professional development, ODE 
staff insisted that such a plan exists. Several times MAP requested a copy 
of this plan. On October 30, 1996, one was faxed to us. 

The plan (as least as it existed at that time) is five pages in length and 
sketchy, at best. It does, in a rather veiled way, endeavor to link state- 
driven professional development efforts with state education reform 
efforts. The document states that the objective of the plan (the plan is 
actually labeled a "systems framework") is to "... guide ODE in providing 
professional development to schools and districts implementing high 
content and performance standards as part of district school improvement 
and consolidated state plans." 

This language is confusing. One would assume that all schools in the 
state are implementing "high content and performance standards" as that is 



the crux of the Educational Act for the 21 st Century. There is in the 
document, however, no explicit reference to the reform law. 

The document consists of a set of "action plans" geared around 
developing "cross-agency work groups to design professional development 
efforts," "utilizing research-based best practice...," "provid[ing] a systemic 
improvement model for professional development," and "creating] 
professional development calendars...." The columns of the plan labeled 
"resources needed," "assigned to," "target date," and "evidence" are all 
blank. 

Perhaps this an evolving plan of recent vintage. But MAP's efforts to 
obtain any previous plan that might be displaced by this one were in vain. 
ODE's efforts to develop a systemic vision for professional development 
were perhaps best summed up by the leader of a curriculum association 
who told us, "If ODE has a strategic plan. I'd love to know what it is." 

To broaden its assistance reach, ODE has established a network of 
regional representatives, ODE employees who are assigned to work with 
districts and schools in a designated sector of the State. MAP discovered, 
however, that regional representatives do not have a clear conception of 
their role. They do not see themselves, for example, as cooperating with 
districts and schools to build local capacity toward achieving school 
improvement goals. Regional representatives do not have a sense of 
mission to guide them to take initiative to motivate or lead local efforts to 
enhance capacity. 

The regional representatives with whom MAP spoke describe their role 
as "being helpful." They seem to view themselves as general purpose 
problem solvers who respond to local requests. They, too, describe 
technical assistance primarily as advice on completing applications and 
interpretations of laws and regulations. 



Over the next several months (until July 1, 1997) a focus of ODE 
technical assistance is assisting districts to complete their Consolidated 
District Improvement Plans (CDIP). Here, we believe, is an opportunity 
missed. A CDIP is a worthy idea. It offers districts a way to think 
coherently and cohesively about the kinds of coordinated efforts they will 
engage in to enhance student learning. At least that is the theory. Practice 
is a little different. 

ODE's specifications for the CDIPs are that districts should, or should 
assist schools to: 1) collect and analyze relevant data about, for example, 
student demographics and student performance; 2) establish measurable 
school-based goals; 3) develop "action plans" consisting of strategies and 
activities designed to improve student learning; and 4) develop an 
evaluation design, including indicators to be used to assess student 
progress. 

Here is a perfect opportunity for ODE to use this tool, the CDIP, as a 
means to help build local capacity— to assist districts and schools to 
develop plans based on the State's vision for education reform, help schools 
identify strategies based on "best practices" and tailor those strategies to 
the needs of individual schools, and link schools and districts with 
appropriate assistance so that the plan becomes a living, breathing, 
working document. Instead, ODE attention is directed toward ensuring 
that districts complete the paperwork properly. The hard task is not to 
write a plan; districts are well practiced at this activity. The difficult part is 
connecting a locally generated plan to the State's education improvement 
vision and strategies. One wonders, for example, why there is no mention 
in the CDIP of Oregon's new student assessments. 
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FINDING #10: 



ODE generally is not viewed as a resource by districts and schools. 

ODE is not perceived by district or school officials as providing adequate 
assistance to help them to ensure that students meet the achievement 
standards established by the state. When asked, on the MAP-administered 
survey, whether ODE is "a resource there for us to use," only 15 percent each of 
district administrators and school principals replied "yes." Administrators from 
small and rural districts, in particular, regard ODE as "marginal, has little 
impact on the quality of education." 

Moreover, "improving local capacity" is not viewed by ESD and district 
administrators as a high priority for ODE. Less than a third of survey 
respondents (31%) gave ODE a rating of does "well" or "fairly well" on this 
dimension. 

Here a district size split is evident. Administrators from the largest 
districts (10,000 or more students) and the smallest districts (fewer than 
2,000 students) rate ODE higher than do other administrators on the 
developing local capacity dimension. Majorities of administrators in the 
largest and smallest districts believe ODE makes this issue a major priority. 

Neither EDS nor district level administrators believe that ODE places a 
premium on providing professional development opportunities. When 
asked if "organizing and providing staff development opportunities and 
programs" is a major or a minor priority for ODE, more than half of ESD 
superintendents (53%), nearly two-thirds of administrators in large districts 
(63%), and nearly half of administrators in small districts (45%) say ODE 
views it as a minor priority. It should be noted that administrators from 
districts with fewer than 2,000 students think ODE does more in the way of 
organizing and providing staff development opportunities. Nearly half 



(47%) say this is a major ODE priority, and more than half (53%) say ODE 
does a fairly or very good job of it. 

Here we refer the reader back to our earlier discussion of leadership. 
Districts believe they are more or less "on their own" when it comes to 
equipping teachers and administrators to meet the State's new education 
expectations. Those expectations and reality, as we have previously 
indicated, are about to crash headlong into each other. 

Local administrators know large numbers of students will not score well 
on the new assessment, at least in the test's initial years. 3 They are fearful 
that the "blame" for students not achieving at the newly established levels 
will be laid solely at their doorsteps. On several visits to Oregon during the 
course of this study, we were regaled with the story of how when 
mathematics scores on the pilot assessment did not reach the then- 
anticipated board-established level, administrators and teachers were 
publicly taken to task by ODE officials— another example of ODE's "We 
just report, but we're not responsible" philosophy. 

To be sure, ODE does make a token effort to provide technical assistance 
within specific subject areas. For example, in a 32-page tabloid mailed to 
every teacher and administrator in the state in January, 1997 ODE described 
the programmatic elements of the reform and invited anyone with 
questions to call particular specialists named as "curriculum contacts" in 
English/ language arts, second languages, science, social sciences and 
mathematics. Simulating the experience of a teacher following up on this 
advice, MAP attempted to reach each of these individuals via telephone. 

All were out of the office or otherwise unavailable and, in fact, we 
established contact with only one within the first week of trading phone 



3 This is a predictable outcome. Oregon’s assessment is a new kind of test, measuring higher standards of 
achievement. It is to be expected that student scores will not, on balance, be stellar at first. 



calls. The press of other duties made it virtually impossible for ODE's 
"curriculum contacts" to function effectively in that role. 

ODE also appears to have no contingency plan in place to assist districts 
and schools whose students do not perform well on the new assessment. 
Districts and schools would welcome ODE's assistance, but they have little 
confidence it will be forthcoming. 

There are other dimensions of "technical assistance" for which ODE t . 
might reasonably be held responsible (though, in MAP's judgment, none is 
as important as the capacity building function). However, when we look at 
the way in which local educators view ODE on, for example, the dimension 
of assisting districts with financial management, ODE does not receive 
exemplary marks. ESD and district administrators do not believe ODE 
assigns a high priority to this function, nor do they believe that financial 
management assistance is something ODE does very well. 

FINDING #11: 

Oregon's ESDs, subject matter associations, universities, and other 
institutions represent vastly underutilized resources available to ODE. 

A number of organizations, institutions, and agencies in Oregon are 
well-positioned to assist ODE in its technical assistance-capacity building 
function. ESDs, colleges and universities, the Northwest Regional 
Education Laboratory, and many local school districts represent a mother 
lode of staff development resources. 

We focus in this finding specifically on the curriculum associations. It is 
these organizations in particular that can buttress a previously identified 
weakness in ODE's current structure, namely insufficient subject matter 
expertise. 
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MAP interviewed 27 current and former leaders of the curriculum 
associations in the disciplines of mathematics, English/ language arts, social 
studies, science, foreign language/ ESL, vocational education, the arts, 
health, and physical education for this study. Locating these individuals 
proved to be a harder task than we had imagined. 

While the Oregon School Directory, published by ODE, maintains 
information about these organizations, we discovered that something was 
inaccurate in about half the listings— wrong phone number, wrong name 
for the individual in charge of the organization, a listing for organizations 
no longer in existence. While MAP recognizes the difficulty of maintaining 
an entirely accurate listing in a published volume, we were struck by the 
fact that ODE staff were unaware that the information was out-of-date. 

When we did locate representatives for each of the organizations, all 
reported to MAP that they are rarely contacted by ODE — several 
complained, in fact, that try as they might, they are unable even to get on 
ODE mailing lists. Here is a ready-made group of knowledgeable 
practitioners in each of the major subject disciplines and yet, say the 
representatives of the associations, they are not part of ODE's professional 
development web. In fact, they say, communication with ODE has all but 
broken down. 

As one association representative told us, "It behooves ODE to have 
communication with the curriculum organizations. If they don't have it 
through a [department-employed] specialist, they need to figure out some 
other way to establish it. But I quite honestly don't understand how they 
[ODE] can serve educators in Oregon without having people responsible 
for the various curriculum areas." 
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FINDING #12 



ODE invests little time and few resources in professional development for 
its own employees. 

If ODE employees are to lead and assist, they must understand the 
content of the reform (not merely the letter of the law) and the nature of the 
change process. While the specific form of professional development must 
be customized to the needs of individuals and groups and specifically 
related to designated roles, all ODE staff must internalize the State's 
conceptual model for education improvement. They must be 
knowledgeable about the complexity, and the long-term nature, of 
institutional change. They must be fully cognizant of the specific standards 
that are to be met and alternative strategies available to accomplish this. 
And they must understand the inter-connected nature of all the 
components of systemic change— curriculum, instructional materials, 
teacher training, assessment, and consensus building. 

Employees must be given time for reflection and opportunities for 
interaction with colleagues. And they must have available to them advice 
and counsel from an experienced mentor, someone who has faced a similar 
situation before. 

In 1994 private employers in Oregon reported that 5 percent of their 
employees' time was devoted to on-the-job training. Most ODE employees 
report than none of their time is spent in activities designed to improve 
their own professional performance. 



RESEARCH AND ANALYSIS 

The ability to understand, interpret, and use data is a critical function 
for ODE. There are at least two important points to be made here: 1) data 



not collected cannot be reported, and 2) data do not necessarily speak for 
themselves. 



FINDING #13: 

As Oregon continues to move to a performance-based system of 
education, ODE's ability to analyze data will become ever more crucial. 
However, ODE currently possesses little capacity to conduct research and 
analyze data on key issues such as student achievement and school 
finance. 

Research and analysis have historically not been responsibilities of the 
Oregon Department of Education. In times past, these functions may not 
have been critical for the agency's productive operation. When schools 
were locally financed, ODE had little need to equip state policy makers 
with data on the relative costs of various kinds of education undertakings 
or to simulate the costs of proposed new programs, or assess the ways in 
which districts allocated their dollars and the kinds of results that came 
from those allocations. These were responsibilities of local fiscal staff who 
were accountable to local school boards. With the enactment of Measure 5, 
all that changed in Oregon. Now, school finance is a state function and the 
agency responsible for the education program must play a more activist 
role in this arena. 

Likewise, when there was no comprehensive statewide program of 
education reform, tied to state-established standards and state-developed 
assessments, districts and schools largely assumed the task of analyzing 
and reporting local achievement scores. These were issues of community, 
or perhaps regional, interest, but not a statewide policy concern. 

Now the tables have been turned on this dimension as well. State policy 
makers understandably look to ODE for an explanation of achievement 
data— what it is, what it means, and what to do about it. 
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Given that research and analysis have not historically been ODE 
functions, and given the limited resources ODE has to accomplish an 
enormous set of tasks, MAP is not surprised to find that ODE is short on 
research and analytic capability. In fact, ODE officials readily admit that 
the agency is a spectator in the school funding process. Making the case for 
level of funding largely is ceded to the Confederation of Oregon School 
Administrators, the Oregon School Boards Association, the Oregon 
Education Association, and a few district superintendents. Analysis of 
funding requests or program costs is conducted by the Legislative Fiscal 
Office and the Legislative Revenue Office. When MAP asked ODE officials 
why ODE has not assumed a more activist role in school finance, given the 
shift to a state-funded education system, the response typically was, 
"because we [ODE] have not been asked by districts or the legislature to 
play a different role." 

While ODE has the capacity to report student achievement scores, MAP 
is not sanguine that ODE can explain what these scores imply in terms of 
concrete activities to improve them. For example, the state has established 
a minimum standard for each subject related to the Certificate of Initial 
Mastery (CIM) and has declared the percent of students who will meet this 
standard. For mathematics, 90 percent of students are to reach a 
standardized test score of 239. Analysis reveals a correlation between the 
number and type of mathematics courses in which students enroll and the 
percent of students who reach the target score. 

In Portland, just one percent of students who take only general 
mathematics scored at least 239 on the pilot assessment. But 32 percent of 
students who had taken both algebra and geometry reached that score. The 
number who achieve the desired score rose to 86 percent among students 
who had taken algebra and geometry, and 100 percent among students 
who had completed pre-calculus. These results would suggest that if 
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students have completed four to five mathematics courses by the tenth 
grade (when the assessment is administered), their chances of achieving at 
the target level are excellent; those chances decline with fewer courses. 
Ensuring that all students have the opportunity for the requisite classes, 
and implementing plans to assist those students whose current 
mathematics proficiency does not equip them to reach desired achievement 
levels, is a costly and time-consuming endeavor. 

If ODE could conduct this level of analysis 4 for each district in the state 
(at present ODE has neither the resources nor the capacity to do so), then 
ODE would be able to determine for state policy makers the relative costs, 
in dollars and time, of meeting the standards Oregon has established. In 
the absence of this capability, ODE is at the mercy of a policy system whose 
expectations may be far outpacing what is realistic or even doable. 

Moreover, ODE lacks the capacity to identify school districts with 
existing or potential financial problems. Financial data reported to ODE are 
neither sufficiently standardized nor sufficiently timely to enable the State 
to make a determination regarding school districts' fiscal viability; nor are 
they analyzed for that purpose. Most school districts are adequately 
managed; but it has been the experience in other states that greater state 
assumption of local funding and tighter limits on the ability to raise money 
locally has led to instances of local district "bankruptcy." Insolvency 
frequently results from long-term commitments for construction projects or 
employee compensation based on overly optimistic projections of future 
revenues. Such cases are rarely detected in an annual audit. Frequent and 
detailed budgeting and expenditure data, carefully analyzed, are required. 

ODE plays little role in ongoing efforts to upgrade financial practices in 
school districts, not does it have a contingency plan for potential district 



4 The example cited in this section was provided by Portland School District staff. 




82 



76 



bankruptcy. Consistent with its position on educational program 
performance, ODE seems to believe that it has no responsibility in this area. 
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FINDING # 14: 



On the increasingly important dimension of school level information, 
ODE's current data system is inadequate. 

Policy makers, district administrators and ODE employees were 
consistent in their criticism of the quality of school district expenditure data 
collected and the ability of ODE to forecast enrollments. The current 
accounting rules are relatively ambiguous. When added to the level of 
aggregation of data reported to ODE, they make it virtually impossible to 
make reasoned judgments about expenditure patterns between and among 
local school districts. ODE collects no data, for example, that would permit 
analysis of school level expenditures. Until reliable school level 
expenditure data are collected, it will not be possible to make any useful 
judgments about the costs of specific interventions, let alone judgments 
about their cost effectiveness. 

Finally, reports need to be customized for specific audiences. 

Legislators may be interested in how all schools are performing or how 
Oregon costs for assessment compare with those in similar states. District 
superintendents may be interested in how the performance of schools in 
their district compares with the performance of schools with similar student 
populations. Parents want to know not only what their children know and 
are able to do, but how their children's performance compares with what 
students know and are able to do in schools elsewhere in the state, country 
and world. 

Averages rarely tell the whole story. Reports of district-wide, even 
school-wide, dropout rates or academic performance may mask important 
differences among ethnic or economic groups. Similarly, average 
expenditure data may mask grossly unequal expenditures between schools 



within a district. ODE must be equipped to tell all of these stories 
accurately and completely. 



FINDING #15: 

ODE's current management information system is hobbled by a set of 
interlocking conditions that reduce the utility of the data collected. 

Perhaps key to this finding is that ODE's management information 
system lacks a coherent philosophical underpinning. There is no agency- 
wide consensus on what data are needed and the uses to which these data 
ought to be put. 

Current ODE data systems are structured in a "stovepipe" fashion, with 
data collected and used only by individual persons or functions within 
ODE. Information, in other words, is collected and stored in a manner that 
restricts access to those who gather it. Thus, those who receive the data 
become "gatekeepers" of the information. Some sharing occurs, but lack of 
a centralized data dictionary inhibits efficient, effective data exchange. 

Moreover, the kinds of data ODE collects from districts are largely 
compliance-oriented information— reports that relate, for example, to fund 
distribution, costs for particular categorical programs (e.g., special 
education), and student attendance. Various units within ODE collect data 
at different intervals, some monthly, others quarterly, still others annually. 
Sometimes ODE offices request data from districts and schools on an ad hoc 
basis— a particular need arises suddenly —without checking (or having a 
way to check) if those data already have been previously collected by 
another ODE office. 

According to ODE managers, each of ODE's offices maintains data on 
personal computers in addition to that maintained centrally. These data are 
reported to be duplicative and sometimes at variance with those kept in the 



ODE master database. Perhaps the most troublesome feature of the office- 
level databases is that only a select few individuals know precisely what 
these databases contain and have access to them. 

There is, for example, no easy way to profile the number of special 
education students by handicapping condition, the costs required 
adequately to serve such students, the number of them requiring 
transportation, etc. A similar situation would be true for comparing costs 
of a particular program or intervention across school districts. In addition 
to databases which are not integrated, it was reported to MAP that some 
data are collected but not catalogued, thereby rendering them inaccessible 
to large sectors of the ODE organization. 

Finally, data are collected because they have always been collected. 

Data collection has not been recalibrated to take into account, and even 
anticipate, data needs that arise as a result of the State's larger role in 
financing schools and State policy makers' greater stake in student 
performance outcomes. 



FINDING #16: 

Annual audits represent an underutilized opportunity for appropriate 
State oversight of district expenditure patterns. 

Currently, districts select and contract with local CPA firms for annual audits. 
Although the State Auditor sets certain standards for audits, individual audits are 
not standardized and school districts have little incentive to expand their scope. 

Some other states make far more extensive use of the auditing function by 
requiring auditors to examine ongoing compliance with student attendance, 
compliance with various state and federal program requirements and other 
functions which are not strictly financial. For example, among the compliance 
components California school districts must address are attendance for 
approximately 10 programs, state incentive programs for longer school day and 



year, various state and federal categorical programs, and state residency. Other 
states have found this practice to be a cost effective alternative to operating 
without necessary information or to relying on state staff to conduct special 
audits. Many of the compliance items which compose ODE's School Improvement 
Visits can be addressed in annual audits, thus freeing up scarce ODE resources for 
higher payoff activities. 

It is important, however, for the reader to be aware that audits, regardless of 
how extensive, are unlikely to provide sufficiently timely information to identify 
school districts headed for financial insolvency. 

COMMUNICATION 

For an organization in the throes of implementing complex systemic 
change, communication is critical. It is through various communication vehicles 
that ODE can promote the State's education reform agenda, and build consensus 
for standards-based school improvement. 

FINDING # 17: 

ODE, with limited resources, manages to communicate in multiple ways 
with policy makers, educators, and the public. However, ODE does not 
currently have a "feedback loop" which would enable it to gauge the 
effectiveness of communication with its various "publics." 

ODE manages to produce a plethora of written documents, available in 
schools, to policy makers, and to the general public, designed to paint a 
continuing picture of education in Oregon. Moreover, ODE has made a 
considerable effort to strip the jargon from its publications. This has not 
gone unnoticed by local educators who give ODE good marks for 
providing "clear and well written reports, publications, and other forms of 
communication." (It should be noted, however, that just one third, or 33%, 



of administrators in large districts of more than 4,000 students laud ODE 
for its communication efforts.) ODE also is quite effective securing 
newspaper attention for education efforts across the state. 

If there are criticisms to be made of ODE's various communiques, it is 
that sometimes they are not sufficiently targeted to a particular audience 
(parent representatives, for example, complained to MAP that they do not 
always get the kind of information they want) and that they usually focus 
on process and procedure rather than on results. 

Moreover, ODE rarely asks of its various constituents, "How are we 
doing?" Thus, ODE must rely on anecdote and its own intuitive sense of 
the utility of its communications. The lack of some consistent means by 
which continually to check to see how ODE's message is being heard means 
that minor missteps that might be corrected instead have the potential to 
become serious problems. 
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Chapter 4 



RECOMMENDATIONS 



This chapter provides recommendations regarding ways in which the 
Oregon Department of Education might more effectively carry out its primary 
mission, continuous improvement of teaching and learning in the State's districts 
and schools. Recommendations are divided into the same categories as the 
findings— Leadership, Monitoring and Enforcement, Capacity Building, Research 
and Analysis, and Communication. 

In addition to the Oregon-specific findings presented in the previous 
chapter, the recommendations in this chapter are derived from extensive research 
on educational change (Fullan, et.al.). This research suggests the following six 
guidelines for agencies charged with responsibility for implementing complex 
systemic change: 

1. Concentrate on increasing the capacity of other agencies to implement changes. 
In other words, no agency should take it solely upon itself to make change 
happen. Organizations need partners. 

2. Be clear about what is being implemented, and devote time to ensuring that 
cooperating organizations understand what is meant by the change and what 
is expected in terms of results. 

3. Have a comprehensive, but flexible, plan to guide the change process. 

4. Make sure that staff in the agency primarily responsible for guiding reform 
(e.g., ODE) have the opportunity to develop knowledge and competence in 
both the process and the substance of the changes to be implemented. 

5. Focus principal attention on improving teaching and learning. Do not be 
distracted by other non-essential, and often easier, activities. 

6. Understand that complexity and persistence go hand-in-hand. Large-scale 
change is never achieved quickly, easily — or without a certain amount of pain. 
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LEADERSHIP 

The recommendations in this section are designed to assist ODE to 
promote, throughout the state, a common vision of education and to develop 
strategic and targeted activities aimed at fulfilling the promise of Oregon's 
ambitious education improvement agenda. 

RECOMMENDATION # 1: 

The Oregon Department of Education should assume a more activist 
leadership role in improving curriculum and instruction in Oregon's 
schools. 

The Oregon Educational Act for the 21 st Century requires significant 
changes in what teachers teach (curriculum) and how they teach it 
(instruction) if students are to attain the rigorous standards called for in the 
legislation. ODE receives generally good marks from informed observers 
for the manner in which it has carried out the first tasks in the reform plan - 
building a statewide consensus around standards that define what students 
are expected to know and be able to do by subject area and grade level and 
creating assessments that authentically measure student mastery of 
reading, writing and mathematics. (Statewide assessments in other 
essential subject areas are still under development.) 

As MAP has stated elsewhere in this report, statewide assessment is an 
extremely powerful tool for capturing the attention of the field and for 
motivating educators continuously to improve their practice. ODE is well- 
advised to continue investing its resources in developing a technically 
sound and legally defensible set of assessment instruments and procedures. 

Having said that, however, it is unclear how content standards and 
statewide assessments, in the absence of additional leadership from ODE, 



will result in the changes in teaching and learning at local schools necessary 
to produce the student outcomes promised by the education reform act. 
Indeed, as matters stand, the opposite outcome— a failure of many students 
to earn the Certificate of Initial Mastery when it first becomes available— 
can confidently be predicted. 

What, then, should ODE do? At a minimum, ODE needs to: 

A) Articulate a clear vision of what kinds of curriculum, teaching 
methods, and teacher knowledge are likely to produce the desired 
outcomes. This vision must be developed and shared with all who 
have responsibility for the education improvement effort, beginning 
with ODE staff and managers. 

B) Develop a sophisticated understanding of the change process. 
Change is very complex. It involves internalizing new beliefs and 
adopting new behaviors. Change only comes about when those who 
are expected to make the change believe that it makes sense, clearly 
understand what is expected, and are provided adequate resources 
and incentives. Those who have the task of guiding a systemic 
change effort, namely ODE staff, must themselves understand what 
needs to change and how change can be accomplished. 

C) Develop a comprehensive, evolutionary implementation strategy. 
Beginning with the vision, ODE must determine all that needs to 
undergo change and devise a plan to make it happen. Those who 
frame this strategy must be well-acquainted with the day-to-day 
reality of schools. They must understand the dynamics of small and 
large organizations (schools come in various sizes) and they must 
understand that curriculum is not generic. Students learn— and are 
assessed on— reading, mathematics, science, etc. 

D) Understand that education improvement is a team effort. ODE staff 
must come to understand that responsibility for student 



performance results is shared by districts, schools— and by ODE 
itself. ODE should not be an apologist for failing schools; but it 
should view itself as part of the larger education system with shared 
responsibility for its outcomes. 

None of this should be interpreted to suggest that MAP believes the 
State should unilaterally control or direct local implementation of the 
reform law. Quite to the contrary, a great deal of local autonomy to 
experiment and innovate is essential if the implementation of Oregon's 
ambitious education reform program is to enjoy a reasonable chance for 
success. State leadership is necessary to act as a catalyst, not arbiter, for 
change; to mobilize the talents and energy of local educators and 
organizations; to set priorities, recognize and reinforce success, and focus 
assistance where it is needed; and to advocate for appropriate levels of 
resources to get the job done. 

Cost implications: This recommendation is essentially cost neutral. While 
additional resources, appropriately applied, might indeed be helpful, they 
are not critical to the implementation of this recommendation. 

RECOMMENDATION #2: 

ODE should be reorganized to reflect a priority for those functions most 
closely related to improving educational programs as well as to enhance 
necessary communication among key program improvement functions. 

MAP recommends that ODE be reorganized along lines more 
representative of the first purposes of education. Of course, the exact 
nature and scope of any contemplated reorganization is for ODE leaders to 
define. The following organization is presented as illustrative, not 
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prescriptive, of the kind of changes that would satisfy our main criterion: 
that organizational structure reflect organizational priorities. 

A) ODE should create a position of Chief Deputy Superintendent (or 

at least Deputy Superintendent for Education Programs). This 
individual, who would be responsible for leading all education 
functions and programs, would report directly to the Superintendent 
and would be in charge of implementation of education reform. At 
the very least all offices except those related to finance and 
administration and State Board liaison would report to this 
individual . 5 Under such a scenario the Superintendent would have 
more time available for dealing with the public and political 
environments. The Chief Deputy would focus on ensuring that 
ODE's programs and activities are coherent and calculated to 
encourage improvement of teaching and learning in local schools. 

The Chief Deputy should be a recognized educational leader with 
significant experience managing large educational enterprises. In 
many ways this individual would possess most of the attributes one 
seeks in an appointed chief state school officer. However, because of 
the public and political demands on the head of an agency, even if 
the chief were appointed, it is likely that there would be a need for 
an experienced educational leader to play a prominent role within 
ODE. 

It is unlikely that ODE's current salary structure would attract an 
educator of the caliber contemplated for this role. Therefore, it 
would probably be necessary to retain this person through special 
contract arrangements or other procedures to allow ODE to offer a 
competitive level of compensation. 

5 A strong case could be made for all offices reporting to this individual, considering that all functions of a 
department of education should support and be subordinate to improving education programs. 
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ODE could be reorganized around the following functions: 
Curriculum and Instruction, Assessment, Capacity Building Special 
Needs Populations, Child Developmenl/Child Nutrition, Monitoring 
and Enforcement, Fiscal/Intemal Management, and State Board 
Liaison. Staffing levels and resources assigned, in each case, should 
reflect relative priorities. 

Curriculum and Instruction — This working group, in 
collaboration with curriculum associations, local educators and 
others, would be responsible for conceiving and implementing 
strategies for improving curriculum and instruction in each 
discipline and at the various developmental levels. Individuals 
assigned to this unit would coordinate the development of 
curriculum documents and organize textbook reviews, synthesize 
current research about promising practices, and generally serve as 
catalysts for improving student outcomes in their discipline. 

It is here that MAP refers specifically to the need for ODE to 
rebuild its subject matter expertise, not so ODE can position itself as 
the font of all discipline-specific knowledge, but so that ODE is able 
intelligently and persuasively to leverage the authority of the many 
experts throughout the state. At a minimum, ODE-level subject 
matter expertise should be available in the disciplines that are the 
included in the statewide assessment. 

Generally speaking an ODE staff member in this unit would have 
gained a sufficient level of expertise through graduate level 
education and extensive experience teaching and/ or providing 
school district level leadership in the discipline. Comparable jobs 
would be school district curriculum directors or managers. In 1995- 
96 the average salary for such persons in Oregon school districts 
with ADM 3000 or more was $66,659. 



Included in this unit additionally would be individuals 
knowledgeable about and responsible for those school-to-career 
efforts that complement and reinforce traditional academic 
disciplines, as well as other programs that are primarily instruction- 
based, such as those funded to reduce substance abuse. 

Funding to support the activities of this unit most likely would be 
primarily state general fund or lottery money. However, Title 1, 
Perkins, Eisenhower and other federal funding would be 
appropriate under certain circumstances. Funds for various 
substance abuse education programs may also be appropriate, 
depending on where ODE decides to house those programs. 

Assessment — This unit would oversee the development and 
coordination of the administration of statewide assessments. 
Working with curriculum experts in C&I and in the curriculum 
associations, the assessment group would implement statewide tests 
that not only measure how well students are performing, but which 
reinforce state-sponsored instructional improvement strategies. 

A word here about collaboration: Each time functions are divided 
into separate units, coordination and articulation problems arise. 
Collaboration takes time and energy. It is a natural tendency in 
organizations for individuals to define their responsibilities in such a 
way as to minimize the need for interacting with others to get work 
done. Each line drawn represents a potential barrier to effective 
collaboration. 

Therefore, each case should be evaluated to determine if the 
advantages of creating separate units outweigh the increased costs of 
collaboration. Separating curriculum and assessment is a 
particularly difficult call to make. The value of both is diminished 
when they are not tightly linked. There may, in fact be, potential 



legal vulnerabilities if what is assessed is not specifically what is 
taught. Thus, for purposes of illustration, assessment has been 
described separately. It could, however, be combined in a single unit 
with Curriculum and Instruction. 

A second program function that would appropriately housed in 
this unit would be program evaluation. Which programs are cost 
effective and "value added"? Which efforts represent promising 
practices that might be adapted by larger numbers of schools and 
districts? Which undertakings ought to be abandoned because their 
payoff is too slight? These are the kinds of questions this unit would 
be equipped to answer. 

Capacity Building —This group would be responsible for 
working with local educators to organize and motivate regional 
capacity to provide technical assistance for improving classroom 
instruction. Technical assistance in this case would go far beyond 
merely advising how to complete plans and applications or 
clarifying the provisions of a law or regulation. Instead it would be 
high payoff assistance such as sustained professional development 
aimed at improving curriculum and instruction. (See Chapter 3 
under Capacity Building for a discussion of the nature of effective 
professional development.) 

Generally, this group would not provide any direct assistance. 
Instead, individuals assigned to this unit would work with school 
districts, ESDs, universities, professional organizations and 
curriculum associations to develop the capacity to deliver service to 
local educators. They would serve as catalysts, motivators and 
entrepreneurs— and would have a stake in the success of every 
school in the region they serve. 
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At a minimum these individuals would be well versed in the 
theory and practice of educational change. That is, they would 
appreciate the complexity and difficulty of changing a system as 
large and diffuse as education. They would understand that change 
comes only as individuals, teachers, administrators and others 
construct meaning from the vision articulated by ODE. They would 
appreciate that, at best, laws are enablers of change, but are always 
insufficient to stand alone as agents of change. They would 
understand that developing a plan, no matter how elegant, is 
peripheral to real change; that change comes only after hard work 
sustained over several years. 

The functions ascribed to ODE's current regional specialists are a 
good starting point for developing the specifications for these 
positions. What we have in mind here are roles that are much more 
entrepreneurial, more clearly focused on improving instruction; in 
other words, initiators rather than reactors. 

Special Needs Populations — Fundamental to the mission of this 
unit is a belief that categorical programs exist to enhance the success 
of eligible student populations in the regular academic program. In 
fact, categoricals are primarily funding sources, not programs per se. 
Consistent with this premise, this unit would house the policy 
making and administrative functions necessary to operate the 
various categorical programs. To the extent feasible, resources from 
these funding sources would be redirected to support specialists in 
Curriculum and Instruction and Capacity Building who would be 
charged with developing strategies for improving the performance 
of eligible students in reading, mathematics, history, etc. The Ed Flex 
provisions associated with federal funding provide ODE ample 
opportunity to use federal money to support the these functions. 
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Child Development/Child Nutrition — The functions of this unit 
are largely conditioned by federal funding sources; however more 
attention than currently is given to articulation with various K-12 
programs, especially special education, may be indicated. 

Monitoring and Enforcement — This unit would have as its 
primary goal streamlining the monitoring process, including school 
improvement visits and other such activities, aimed at ensuring that 
school districts adhere to various state and federal laws and 
regulations. This unit should be very lean and directed only at those 
monitoring and enforcement activities that are essential to matters 
such as state and federal laws pertaining to health, safety, and civil 
rights. 

Fiscal and Internal Management — This office would be 
responsible for an upgraded ODE capacity for research, analysis and 
advocacy in school finance. It would also house internal 
management functions such as personnel and accounting. 

State Board Liaison — As is currently the case, this office would 
continue staffing and supporting the activities of the State Board of 
Education. 

The organizational structure suggested above should be viewed 
as merely illustrative of one that communicates to employees, 
clients, and constituents that it places high value on teaching and 
learning. It is not offered as a specific proposal for the way in which 
ODE should be organized. Reorganization should grow after 
refinement and clarification of the agency's mission and careful 
consideration of alternatives. 

A final paragraph about reorganization. It is not unusual for 
agencies to reorganize simply to convey the impression of progress or 
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provide the imprimatur of a new leader. Such changes are often no 
more than cosmetic and give rise to the old saw about rearranging the 
deck chairs on the Titanic. Reorganization is costly in time and 
emotional energy and should be attempted only when there is potential 
for enhanced effectiveness. MAP believes that for ODE, the potential 
benefits of structural reorganization far outweigh the costs. Even so, 
reorganization by itself will have little effect without attention to 
alignment of mission, employee training, qualifications, compensation, 
and recommendations offered elsewhere in this report. 

Cost Implications: The principal immediate new allocation required to 
implement this recommendation is for the salary of the Chief Deputy. 
These positions typically command $100,000 to $120,000 per year in 
state departments of education. Beyond this new allocation, every effort 
should first be made to find resources within the existing organization 
to cover the costs of new or added positions. Once this effort has been 
made, ODE will be in a better position to make the case for the "real" 
number of dollars necessary for an appropriately staffed new structure. 
(It should be noted that the structure suggested here by MAP is simply 
illustrative; it may not be the final organizational arrangement on which 
ODE settles.) Additionally, consideration should be given to filling new 
positions over time so that any added costs can be incorporated and 
absorbed in a gradual fashion. 

RECOMMENDATION #3: 

ODE should develop for every organizational unit annual workplans 
with measurable outcomes and budgets specified. Changes to these 
plans should be negotiated when indicated by changing priorities. 

Managers should be evaluated, at least in part, on the basis of how well 
their performance measures up against approved plans and budgets. 
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Often the process of planning is at least as valuable as the product. It is 
in the give-and-take of planning that the leadership communicates 
priorities and expectations and that managers find opportunities and make 
commitments to collaborate. It is also during this process that redundant or 
lower priority activities can be discovered and placed in their proper 
perspective. 

This planning process must be comprehensive and systematic to ensure 
that it accurately reflects a coherent vision for the entire agency. Individual 
unit plans, in other words, must combine to form a coherent whole. 

If this process is to succeed, the Superintendent and ODE managers at 
every level must be fully engaged. Without their participation, the 
planning process quickly will come to be viewed as a "make work" 
exercise, and those charged with its completion will not pay sufficient 
attention to it. Genuine engagement of the leadership in a systematic 
process of planning can do much to remediate the fragmentation and loose 
coupling evident currently in ODE. 

Managers cannot reasonably be held accountable for that over which 
they have little control. This is particularly true in the area of resources 
managers are expected to manage. Managers and other staff members who 
know clearly what is expected of them are likely to be most effective when 
they are given the resources necessary to accomplish those expectations 
and are held accountable for results. 

MAP thus recommends additionally that each ODE unit's workplan be tied 
to a budget and that each manager be held accountable for accomplishing the 
agreed upon scope of work within the bounds of that approved budget. Changes 
to a budget or a workplan should be negotiated with the manager with sufficient 
perspective to ensure that such changes are consistent with overall ODE priorities. 
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Cost implications: This recommendation is cost neutral. 

RECOMMENDATION #4: 

The Oregon Department of Education, in cooperation with the legislature 
and appropriate state agencies, should take the steps necessary to attract 
and retain education professionals with sufficient expertise, experience, 
credibility and stature to provide the leadership necessary to implement 
the Educational Act for the 21 st Century. 

Lagging salary levels and civil service regulations make it difficult for 
ODE to attract and retain education professionals who command respect 
among their peers. If ODE is to influence and gain the respect and 
cooperation of local educators, its representatives must be seen as superbly 
qualified in their own right. As we have stated previously, ODE needs a 
cadre of specialists who have experience at least equivalent to a school 
principal or district curriculum administrator in a medium to large district. 
And ODE needs as part of its own staff individuals who are recognized 
experts in their subject matter disciplines. To attract and retain such 
persons will necessitate that they be paid a salary comparable at least to 
their district peers. 

On the other hand, many of the administrative and enforcement 
functions that ODE performs can be discharged by persons without a high 
level of expertise in education. This suggests the need for dividing the 
specialist class into two levels, the higher paid of which would be for the 
professionals described above. It also introduces the possibility of pursuing 
a "cost neutral" transitional strategy of creating and filling the higher level 
positions only as current positions become vacant. This would make 
possible systematic enhancement of the skill level of ODE's workforce 
without a significant increase in salary expenditures. 
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Cost implications: As indicated above, implementing this recommendation is 
potentially cost neutral, depending on what resources, if any, can be "captured" 
from ODE's current budget. However, over time, added ODE appropriations are 
likely to be necessary, depending on the final organizational alignment on which 
ODE settles. In general, based on the current market, a new salary schedule for 
top level ODE employees should be constructed as follows: deputy 
superintendent salary should be pegged at $90,000 to $95,000 per year; assistant 
and associate superintendents should earn in the range of $75,000 to $80,000; 
specialists should be paid $60,000 to $75,000. 

RECOMMENDATION # 5: 

Oregon's elected position of Superintendent of Public Instruction should 
be replaced with an appointed position. 

The trend in states in this nation has been away from elected and 
toward appointed education chief executive officers. At the beginning of 
this century, more than 70 percent of state superintendents were elected. 

By 1996 just 15 states maintained elected chief state school officers. 

A primary reason for the move from elected to appointed state 
superintendents is to ensure that accountability for performance of the 
education system rests with the executive who controls the budget. As 
states have assumed ever greater responsibility for education dollars, they 
need likewise to assume greater accountability for how those dollars are 
spent. 

In Oregon, ODE, and thus its chief executive officer, is responsible for 
education performance. Yet ODE has virtually no authority over the 
dollars that are allocated to or disbursed for education purposes. Thus, a 
superintendent who is actually part of the executive branch which 
generates the budget makes considerable sense. 
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MAP is aware that the issue of changing to an elected superintendent 
has been raised before in Oregon and each time the idea has died. While 
we are not in a position to comment on the disposition of the Oregon 
electorate's views on this matter, we can provide some insight into the 
thinking by ESD, district and school administrators. 

When asked, on the survey prepared for this study, "Do you think it 
would be better if the state superintendent of schools were appointed by 
the governor or elected to office?," 57 percent of ESD administrators, 72 
percent of administrators in large districts, 64 percent of administrators in 
small districts, and 54 percent of principals replied, "appointed by the 
governor." 

States have accomplished the changes described above in a variety of 
ways. Some have implemented constitutional or legislative changes that 
eliminated the elected office and created an appointed one. Others (e.g., 
Kentucky) have accomplished the same purpose de facto by simply 
establishing an appointive office and shifting the resources to do it. 
Oregon, thus, has options available to it should it choose to make this 
change. 



Cost implications: The principal cost to be incurred here is in the salary of 
an appointed Superintendent. The annual compensation range should be 
$125,000 to $150,000, less the salary currently paid. (As a point of 
comparison, the Superintendent in Portland earns $150,000 per year.) As an 
offset, if there are particular perquisites that attach to constitutional officers, 
these should be considered as potential dollars to be saved. 
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MONITORING AND ENFORCEMENT 



The Oregon Department of Education is making efforts to transform itself 
from an organization whose main function was ensuring that applicable program 
requirements were being observed to an organization dedicated to building local 
capacity to meet mutually agreed-on education goals. These recommendations 
are designed to encourage and promote that course of action. 



RECOMMENDATION #6: 

Conventional kinds of monitoring and enforcement activities should be 
streamlined and reduced. 

There are some federal and state requirements that ODE must monitor. 
Rules to protect the health, safety and civil rights of children come quickly 
to mind. Monitoring for compliance should be kept to a minimum, 
however, not because it is inherently bad, but because ODE's limited 
resources are urgently needed to provide leadership for higher academic 
performance. Monitoring inputs rarely influences children's academic 
achievement. Wherever possible, responsibility for monitoring should be 
delegated to local administrators or contracted out to Education Service 
Districts or other responsible parties. 

In recent years, ODE has put much more emphasis on providing 
technical assistance at the local level even while enforcing compliance. 
Evidence of this evolving change (as reported by ODE's clients) is seen in 
the slowly shifting emphasis in the school improvement visits. However, 
more is required. 

MAP recommends that activities that detract from ODE's central 
mission of improving teaching and learning no longer be statutorily 
mandated obligations of ODE. In particular, we refer here to criminal 








98 



background checks and the three-year required cycle of school 
improvement visits. 

The law requiring that ODE be responsible for the paperwork connected 
to criminal background checks of non- certified employees should be 
changed. ODE should not be burdened with this responsibility. 

MAP also recommends that the statute which mandates school 
improvement visits to every district every three years be amended. There 
are a number of problems with the current law. First, the premise that 
enforcement of various laws will bring about substantial improvements in 
student performance in not supportable. After extensive research on 
educational change in the United States and Canada, Fullan concludes: 
High regulation and monitoring can achieve minimal compliance 
at best. This may be necessary in situations that are so bad that 
the most basic conditions do not exist, but regulatory approaches 
cannot accomplish basic reform. Laissez faire, leave-it-to-the- 
districts orientations are also not the answer. The research we 
have reviewed... strongly suggests that low to medium regulation 
(guidelines more than prescriptions), combined with high 
engagement (negotiation, technical assistance, monitoring, 
feedback, problem solving) works better . 6 

Second, the sheer volume of these visits swamps ODE's resources and 
diverts too many people from other activities with substantially higher 
potential for improving teaching and learning. A third point, related to the 
second, is that the current law requires ODE to treat all districts the same, 
without regard to their performance. As a consequence, a district with 
serious problems is no more likely to be visited than one that is excelling. 
Finally, the teams conducting the visits are composed predominately of 



6 Fullan, Michael G. with Suzanne Stiegbauer. The New Meaning of Educational Change . NewYork: 
Teachers College Press, 1991. 



ODE employees who are unlikely to have recent experience in a K-12 
educational setting. 

MAP recommends that the law be changed to address all four of these 
concerns. The first step would be a careful review of the laws regulating 
schools and elimination of those provisions that are inconsistent with 
Oregon's avowed philosophy of local control. Only those statutes that 
relate to health and safety, stewardship of public moneys and those that 
have a demonstrable relationship to facilitating a high quality instructional 
program should be retained. 

A more cost effective law would be one that assumes that school 
districts are in compliance unless there is some indication otherwise. 

Rather than sending a team of ODE employees every three years to check if 
a district has adopted goals, has adopted a plan to teach about infectious 
diseases or has a coordinated guidance program or goals for media 
instruction, the state could require districts to certify that they are in 
compliance with applicable laws. The need for site visits could be largely 
replaced by instituting a time-bound, systematic complaint procedure, in 
which ODE would investigate and resolve complaints brought by parents, 
students, school district employees and other citizens. 

Based on the premise that districts are in compliance unless there is 
evidence to the contrary, the law should establish criteria to identify which 
districts should be visited. The two most important criteria, and perhaps 
the only two, are student academic performance and financial management. 
If a school district's students demonstrate high achievement and annual 
audits indicate that the district is solvent, there does not seem to be any 
compelling justification either for expending ODE resources to send a 
visiting team or spending the district's resources to prepare for the visit. 

When test scores and perhaps other measures indicate that student 
achievement is unacceptably low, or an audit reveals serious financial 
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concerns, ODE should organize a team of experts who can provide valuable 
assistance related specifically to the identified problems. With few 
exceptions, ODE employees will not possess the necessary specialized and 
technical expertise. For this reason, MAP recommends that the law make 
provisions to allow ODE to call on educators in local districts and ESDs as 
necessary. 

MAP is aware that the laws governing various federal programs require 
some level of monitoring. Even these rules are in a state of flux as the 
federal government moves toward more state and local flexibility. MAP 
recommends that ODE negotiate with the U.S. Department of Education 
monitoring procedures that are as consistent with the law as possible, while 
maintaining for ODE the necessary level of flexibility. 

ODE should also undertake a review of its own internal procedures and 
regulations. The goal of this effort would be to eliminate or simplify those 
activities that divert resources from ODE's primary mission. 

Oregon is well positioned to make other changes as well. The 
opportunity to reduce federal regulation already exists because Oregon has 
been designated an EdFlex state. The State Board of Education has the 
authority to waive federal regulations in many areas and should do so 
liberally. More critically, though, the mindset of ODE's main job as helping 
local districts and schools develop the capacity to educate students well, as 
opposed to regulating local behavior, needs further to permeate the 
organizational culture. 

Cost implications: This recommendation may well represent a cost savings. 
Any resources saved should be redeployed to other ODE activities and 
functions. 
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RECOMMENDATION # 7: 



Essential monitoring and enforcement activities should be consolidated 
into a single,, relatively small unit of ODE. 

Monitoring and compliance activities required by federal programs and 
for compliance with federal and state health, safety, and civil rights 
requirements should be consolidated into a single ODE unit. This unit 
should devise a simplified reporting system that minimizes the time and 
resources required of local district and school personnel. 

By organizing essential monitoring activities into a single unit 7 , the data 
collected from schools and districts can be centrally aggregated and 
rendered optimally useful. Consolidation of regulatory activities will also 
assist ODE to develop an integrated management information system and 
assure increased sharing of information among program managers and 
staff. 

Cost implications: Implementing this recommendation will not cost money 
and, in fact, may result in a savings. Any dollars that are saved should be 
redeployed within ODE for other functions. 



CAPACITY BUILDING 

Building the capacity of districts and schools to assist students to achieve at 
the new, higher state-established standards is one of the most critical functions for 
ODE. It is all the more critical given that a recent study reported that 59 percent of 
Oregon's teachers have out-of-field teaching assignments (See special issue of 



7 The exception here would be financial monitoring which will need to be housed in the finance 
section of ODE. 
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Education Week ) and large school districts, such as Portland, have eliminated their 
curriculum departments entirely in the wake of budget cuts. 

RECOMMENDATION #8: 

The Oregon Department of Education should strategically redeploy the 
resources currently being used for "technical assistance" in Oregon to 
create multiple networks of service providers across the state. The role of 
ODE should be to serve as a catalyst and clearinghouse for new, strategic 
service provision efforts. 

One of the most important responsibilities of the Oregon Department of 
Education is ensuring that educators in Oregon have the knowledge and 
skills to accomplish the kind of challenging program called for by the 
Educational Act for the 21 st Century. In order to make standards-based 
education reform a reality, teachers and other educators will need to know 
how to conduct performance-based student assessments, implement 
innovative curricula, and develop a suite of new professional skills and 
instructional strategies. 

Acquiring new knowledge and skill is not a one-time proposition. 

Districts and schools need to develop the capacity for continuous 
improvement. They must be able to evaluate the extent to which they are 
carrying out the State's mandate to educate all students to high standards 
and to make mid-course corrections in areas in which they find their efforts 
are falling short. 

The most efficient and likely most effective means by which ODE can 
assist schools and districts on this dimension is not by acting itself as a 
direct provider— its own human and fiscal resources are too few— but 
rather by: 1) serving as the catalyst, convener and organizer of the wealth of 
professional development talent available in the state and, 2) connecting 
educators with available services. Education service districts, curriculum 
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and other professional associations, within-school-district experts and the 
Northwest Regional Education Laboratory are examples of providers who 
can offer the kinds of professional development opportunities schools and 
districts require. ODE should take responsibility for developing networks 
of these service providers, helping to coordinate their activities, extend 
their geographic reach, and control costs. 

Perhaps the reason that professional development is so routinely 
overlooked in education is that it is not cheap. Effective professional 
development has to be specific to the participants' jobs; it has to have 
sufficient duration for participants actually to develop new skills, and it has 
to be reinforced over time. For teachers, experience suggests that at least 
three to six weeks in a full-time learning environment followed by at least 
two years of follow-up activity is required to make a lasting change. 

Similar experiences are necessary for administrators. 

Few states are able or willing to add the amount of resources such 
massive training would cost, nor is it entirely necessary. Thus, before 
adding resources, ODE should look for existing resources to redirect. The 
first candidates for such redirection would be any money being spent (by 
ODE or by districts) on short-term, "parachute-drop" workshops which 
offer a negligible payoff in changed instruction. Second, funds expended 
for activities that benefit one or just a few individuals should be considered 
for redirection to activities with a greater system-wide payoff. Categorical 
program budgets are often good sources for redirecting funding to high 
pay-off activities for teachers of eligible students. 

The greatest payoff is likely to come when Oregon educators, working 
collaboratively, develop system-wide strategies for improving curriculum 
and instruction. Such subject-specific strategies would call on all the 
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resources in the system and target them in a manner calculated to create a 
critical mass of improved instruction. 

Cost implications: The first priority should be for ODE to leverage existing 
resources in ODE budgets (e.g., federal dollars targeted for professional 
development) and in local district budgets. A promising model of staff 
development is the California Subject Matter Projects (see Chapter 6 for 
more detail on this effort.) Using the California Subject Matter Projects as a 
rubric, and assuming that 5,000 Oregon teachers participate each year in an 
intensive 3- to 5-week institute, with follow-up, Oregon can anticipate 
annual costs of $2 million to $3 million. Statutory authority, and dollars, for 
an Oregon version of an effort such as the California Subject Matter Projects 
appears to exist in ORS 329.745 which sets aside funds for professional 
development centers. 



RECOMMENDATION #9: 

The Oregon Department of Education should assign a much higher 
priority to the professional development of its own staff. 

As matters stand, there is no discernible provision for professional 
development built into the day-to-day operations of ODE. Although the 
agency is not so large that a good deal of information cannot be exchanged 
"in the corridor" on an informal basis, the lack of a structured vehicle for 
promoting interdepartmental communication takes its toll. Sometimes it 
means that the left hand does not know what the right hand is doing. 
Sometimes it means that specialists go through the motions of executing a 
task without really understanding how it fits into the big picture. If ODE is 
to lead the state education community, it needs to have its own house in 
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order in terms of fostering the understanding and commitment of its 
employees to a shared vision of education excellence. Such a commitment 
does not just happen; it must be cultivated. 

The first priority of any organization should be to ensure that every 
employee understands the organization's mission and how that employee's 
job fits into that mission. Employees charged with implementing complex 
programs such as the Educational Act for the 21 st Century should 
understand at least as well as those they are trying to help what is 
necessary to implement the program. This implies a knowledge far deeper 
and more detailed than the mere provisions of the law. It requires an 
understanding of the dynamics of organizational change and concrete ideas 
of what kinds of curricular and instructional changes will be necessary. 

ODE staff must have the time and opportunity to develop the requisite 
skills to enable them more effectively to do their jobs. 

Cost implications: It is difficult to estimate the actual cost of implementing 
this recommendation. As we indicated in the Findings chapter of this 
report, private employers in Oregon reported in 1994 that 5 percent of their 
employees' time is devoted to on-the-job training. Thus, the real issue here 
is making time for staff development a part of the job of ODE employees. 

RESEARCH AND ANALYSIS 

MAP has referred frequently in the Findings chapter of this report, as well 
as in previous recommendations, about the need for ODE to have the capacity for 
research and analysis. The next set of recommendations speak specifically to the 
kinds of research and analytic skills ODE would find the most useful. 
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RECOMMENDATION #10: 



The Oregon Department of Education needs greatly to increase its 
research and analytic capability. 

Citizens and policy makers in Oregon, not to mention managers of ODE 
itself, need timely, accurate, relevant information bearing on policy options 
impacting education in the state. Implementation of a new ODE database, 
a recommendation detailed later in this report, will provide the basic data 
structure necessary to improve reporting, but will not satisfactorily resolve 
a troubling information gap by itself. The data collected also must be 
organized, analyzed and reported in a way that is useful to potential 
decision-makers. 

The cornerstone of standards-based reform is the statewide assessment 
system. Properly analyzed, student assessments provide valuable clues for 
determining what interventions may be necessary at what grade levels and 
in subject areas in specific districts, schools, or sub-populations of students. 
Research and analysis can assist ODE to identify needs, allocate resources 
appropriately and, ultimately, make choices that improve student learning. 

Increasing ODE's level of effort in the area of research and analysis 
will likely require some other activities to be curtailed. The great benefit of 
an enhanced research and analysis capability is that it will allow ODE 
managers to make such choices from a position of well-informed strength. 

Cost implications: ODE's recently assigned school finance expert has just 
left the agency so there is currently an opportunity for a new hire at no 
additional cost. In general, implementing this recommendation will require 
2 to 3 full-time employees with specific expertise in areas such as finance, 
assessment, and data management. Assuming an annual salary of 








107 



approximately $60,000 plus benefits, the annual cost is estimated at 
between $150,000 and $225,000. 

RECOMMENDATION #11 

The Oregon Department of Education needs to play a more active role in 
developing, collecting, and reporting financial information. 

When most funds for education were raised and spent locally, there was 
little demand for reliable and timely budgeting and expenditure reporting 
at the state level. Now that the State provides most of the revenue to local 
districts and constrains local spending levels, it acquires a far greater 
burden to account to Oregon taxpayers regarding how the dollars are 
spent. 

First and foremost, the State must ensure that local school districts 
employ adequate financial management procedures to guarantee that funds 
are appropriately accounted for and that district expenditures do not 
exceed revenues. Second, the State should have sufficient data and 
analytical capability to predict with reasonable accuracy the cost of 
proposed education system changes (e.g., changing class size, altering 
graduation requirements, requiring a minimum level of building 
maintenance). Finally, the State should be able to analyze expenditure 
patterns associated with identifiable programmatic practices in order to 
make judgments about the relative cost effectiveness of various educational 
interventions. 

ODE is not currently well positioned to perform any of these functions. 
MAP was repeatedly informed that accounting data are not standardized 
across districts. While all districts follow the state-prescribed accounting 
manual, there remains sufficient ambiguity and room for local 
interpretation so as to render inter-district fiscal comparisons nearly 
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meaningless. Thus, MAP recommends that ODE assign high priority to 
standardizing accounting procedures. 

Undertaking this task is laborious and time-consuming. It cannot be 
done by legislative fiat, nor unilaterally by ODE. Standardizing accounting 
procedures requires buy in from and understanding on the part of local 
administrators, business managers, and ultimately, the account clerks who 
enter the data. As described elsewhere in this report, the appropriate role 
for ODE would be to convene, motivate, facilitate, and assist, but otherwise 
to participate as only one of several users of these data. The legislature 
should provide appropriate funding to underwrite this process. 

MAP also recommends that simultaneous with standardizing 
accounting procedures, ODE, along with other relevant state agencies, 
revise the financial reporting procedures so that district budgets are 
received and analyzed before the beginning of each school year and that at 
least one mid-year expenditure report is submitted to and analyzed by the 
State. ODE will need to enhance its capacity to analyze the reports as they 
are submitted. There are presently too few ODE employees with the 
expertise adequately to meet the relatively modest current demands in this 
area. Thus, it would be necessary for ODE to hire additional employees 
with the requisite experience and skills. Enabling legislation may be 
required as well. 

In addition to the capacity to detect school districts with potential 
financial problems, ODE should take anticipatory action calculated to 
prevent such problems and develop contingency plans for dealing with 
districts which find themselves nearing bankruptcy. MAP recommends 
that ODE review the comprehensive system established by California. 
Chapter 1213 (AB 1200) of the California Education Code was revised to 
codify procedures in order to address reporting, prevention, and 
consequences. One of the more innovative features of this law was the 
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establishment of the Financial Crisis and Management Assistance Team 
(FCMAT). FCMAT conducts ongoing training of school business officials to 
upgrade their skills and minimize the potential of a financial crisis. It 
provides assistance on request, and intervenes in the event of a financial 
crisis. FCMAT is a creatine of the state; however, it is managed and staffed 
by county office of education employees (analogous to ESDs). 

Standardized accounting data and enhanced analytical capacity should 
be sufficient to allow ODE to make reliable projections of various macro- 
system changes. These improvements alone will not be adequate, however, 
to make judgments about cost effectiveness of school and classroom level 
interventions and educational practices. Without reliable school level 
expenditure data, it is not possible to make such judgments. 

Most school districts do not presently collect school level expenditure 
data, and considerable effort, time, money, and training of district 
personnel will be required before these data are reliably reported. 

Moreover, it will require the combined efforts of ODE, the State Auditor, 
other state agencies, local school districts, and others to accomplish this 
task. However, without detailed information regarding how money is 
spent at each school, it is not possible to make reasoned comparisons about 
the relative worth of current or proposed expenditures. Nothing less than 
valid, reliable, detailed information about school level expenditures will 
allow Oregon policy makers to address public education expenditures with 
the creativity and innovation they have brought to health care costs and 
other areas of public service. 

Cost implications: We would estimate that developing the kind of 
standardized data system described above would cost in the $2 million to 
$3 million range. California is in the process of implementing just such a 
system. In addition to developmental costs, that state has allocated a small 
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sum of money ($5 per ADA) to districts to defray implementation costs. It 
should be noted that this sum does not reimburse districts their costs; 
however, districts regularly update their data collection systems anyway 
so, in this instance, as an encouragement to districts, the state is just 
offsetting this cost. Costs of additional ODE employees required to provide 
appropriate data analysis will be approximately $120,000 to $180,000, 
depending on the final ODE organizational structure. 

RECOMMENDATION #12: 

To enhance its Management Information System, the Oregon Department 
of Education should establish a single database of information that is 
accessible to anyone within ODE and that provides a basis for responsive 
answers to requests for information from outside ODE. 

Development of an ODE database is not demanding from a technical 
viewpoint and costs should be recovered by increases in productivity. ODE 
is aware of the need for information that accurately represents student 
outcomes and that makes the linkage to funding and other governmental 
policy decisions in the post-Ballot Measure 5, standards-based education 
environment. It has been studying satisfactory alternatives. One of the 
proposed solutions is installation of a UNIX-based central file server which 
would host ORACLE database software. This approach would provide a 
satisfactory technical solution and would facilitate easy access to ODE 
information. 

Prior to defining its information needs in detail, however, ODE must 
undertake a major philosophical shift in how it gathers and uses 
information. Current systems are structured so that data are proprietary, 
the property, in effect, of those who collect it. Some sharing occurs but it is 
limited at best. A single ODE database and a commonly used data 



dictionary would greatly enhance the availability, and the utility, of 
collected data. 

What are the steps ODE should take? First, ODE must determine which 
data, and at what level of detail, are necessary to be collected and 
maintained in order to address the needs of diverse users within and 
outside of ODE. Among these groups would be ODE managers and 
employees, the Oregon legislature and governor, local school district 
administrators, auditors, parents, and citizen groups. These and others 
should have a reasonable expectation that ODE can provide reliable 
information regarding a broad range of topics without incurring undue 
delay or additional costs. The determination regarding the nature of the 
data collection should not be undertaken unilaterally by ODE. All 
stakeholders, both data providers and users, should participate in making 
this critical threshold decision. 

Next, ODE must develop a data dictionary (a directory of data 
definitions and locations). This effort should be undertaken with the range 
of data providers and users in mind. Without the full cooperation of those 
who are being asked to provide the data, it is unlikely that the product will 
be sufficiently reliable or useful. Similarly, decisions about the definitions 
of data will affect its usefulness to ODE and to its clients. It is rarely 
possible to anticipate all data needs, but careful analysis and extensive 
involvement of stakeholders "up front" will yield substantial dividends in 
terms of credibility and goodwill later on. 

Third, ODE will need to determine the frequency and mode of data 
collection. Mode implies not only the format and media, but also the 
electronic data interchange protocols which will be accepted by ODE. If 
ODE, as planned, acquires a UNIX-based central file and ORACLE 
database, the agency will be able to accept data in a wide variety of formats. 
Over time, a more efficient procedure would be to move toward a 
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standardized format for all data reported to ODE. This may not present a 
significant challenge to many school districts. For example, approximately 
70 school districts currently contract with the Lane ESD for financial 
services and data processing, thus requiring a single set of changes for all 
70 districts. Again, it is essential that the nature and timing of such an 
action be mutually agreed to by ODE and other data stakeholders. 

ODE should move as rapidly as possible to eliminate all hard copy data 
submissions. The time and resources necessary to handle and translate 
paper reports simply are not justifiable. Ideally, data would be transmitted 
to ODE electronically, but mailing disks could be an acceptable substitute. 
The means for electronically transmitting data to and from ODE exist in 
every school district that has a computer modem, and access to the Internet. 

Cost implications: MAP has been informed by an ODE official that ODE 
can use the hardware of another agency, thus eliminating this potential 
cost. In addition, the cost of required software is already included in ODE's 
budget. Training costs will be incurred but only for those select individuals 
who will actually need to run the new system. 



RECOMMENDATION #13: 

ODE and the State Auditor should expand existing local district CPA 
audits to provide more information for policy makers. 

MAP recommends that ODE and the State Auditor examine all 
information currently collected, including data gathered during School 
Improvement Visits, for possible inclusion in the scope of annual district 
audits. Some programs and functions might consistently be audited 
annually; others might be audited on a less frequent schedule or only when 
a particular need arises. Other states have found this means of auditing to 
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be a cost effective method of ensuring adequate local district management 
of certain functions and programs. 

Cost implications: Implementing this recommendation will increase the 
costs of annual local audits. The amount of the increase will depend on the 
degree to which the scope of audits is expanded. Moreover, the state will 
need to determine the extent, if any, to which it wants to offset local district 
audit costs. 



COMMUNICATION 

Communication is key to the successful implementation of complex change. 
ODE has made a good start with its various kinds of communications. This fi nal 
recommendation is designed further to assist ODE on that path. 

RECOMMENDATION #14: 

In promoting a single, unifying vision of what public education is 
striving to become, the Oregon Department of Education should continue 
to adjust its communications to its various "publics" and should develop 
feedback loops capable of measuring the success of its multiple 
communication efforts. 

Communication is a crucial tool for promoting the Oregon education 
reform agenda. ODE has many audiences with which it must 
communicate, including the public, state policy makers and local educators. 
Each of these audiences must understand and actively cooperate in the 
implementation of the State's school improvement efforts. While the 
message to each of these groups is the same, the way that message is 
couched will vary significantly in emphasis and detail. Communications 
must be tailored to various audiences. One size rarely fits all. In effect, 

ODE should have a strategic plan for insuring that: 1) the public has the 
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information it needs in order to understand and support the goals of the 
reform law; 2) state policy makers have the data they need to continue to 
make sound educational policy, and, 3) local educators have the 
information they need in order to successfully implement reform. 

ODE receives reasonably good marks for providing clear and 
frequent communications. Now it must focus those communications less 
on process dimensions and more intensely on the substance of the reform 
enterprise. ODE also needs regularly to apprise itself of the extent to which 
its various communications are meeting the needs of the audiences for 
which they are intended. Toward that end, ODE should establish feedback 
loops that allow it to track the relative usefulness of specific 
communications and to fill in information gaps where such are shown to 
exist. Feedback should be gathered from "clients" on all ODE activities, 
and these data should be gathered and analyzed in a regular and systematic 
fashion. Moreover, data need to be put to use. If a client complains of 
insufficient information, or information which is not "user friendly," ODE 
should come to view this kind of comment as constructive criticism and 
take appropriate steps to address the client's concerns. 

One means by which ODE can establish feedback loops is through a 
series of annual surveys of local educators and citizens. Survey questions 
would be designed to assess the relative effectiveness of ODE's various 
communications. Responses would provide ODE with data by which to 
make judgments regarding appropriate adjustments to existing documents 
or additional communiques needed. 

Cost implications: If ODE were to use statewide surveys to gather 
information about its various communications, the cost is likely to be in the 
$50,000 to $75,000 per year range. 
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Chapter 5 



RESULTS OF LOCAL EDUCATION SURVEY 

Peter D. Hart Research Associates conducted a statewide survey of local public 
school district administrators, Education Service District administrators, and school 
principals in Oregon as one component of the program evaluation conducted by 
Management Analysis and Planning Associates. The purpose of this survey was to obtain 
a view of the Oregon Department of Education from beyond the ODE itself and beyond the 
confines of the capitol and those heavily involved in public education at the state level. 
What the survey provides is a view of the ODE and its role in public education from the 
local public school district perspective and from the perspective of those who are daily and 
directly responsible for the education of Oregon’s public school students. 

This chapter provides a narrative summary and description of the survey results. 
Throughout, “local public school officials” is used as a convenient rubric for the three 
populations from which samples were drawn for the survey, and the term “administrators” 
is reserved for the superintendents, various types of assistant superintendents, business 
managers, and other ESD and district-wide personnel, while principals though also holding 
administrative positions, are referred to directly as “principals.” It should be noted that, on 
balance the educators surveyed give the state reasonably good marks for the education 
system and likewise give ODE positive ratings on a number of significant dimensions . 8 



8 A complete description of the research methodology used for this survey can be found in Appendix A 
Methodology. 
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The State of the State 



To set the stage and begin to understand the local public school perspective, we 
note that while local district administrators’ and principals’ overall evaluation of the 
quality of public education in Oregon (kindergarten through 12 th grade) shows a great 
many solid marks, few of those surveyed are willing to confer the highest marks. As 
Figure 1 shows, no more than one in ten school principals or local district administrators 
would rate the State’s school system below a grade of “B,” but just one in four school 
principals and one in six local district administrators are willing to assign an “A” to the 
quality of education provided by Oregon’s K -12 system. 

Figure 1 

Administrators/Principals Grade the Quality of Public Education in Oregon 




While local district administrators and school principals in the larger districts are a 
bit more upbeat, the clear consensus among these local school officials is that the state is 
doing a good job of educating its young people, but that room for improvement exists 
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nonetheless. Yet. there is nothing in these results to indicate that more than a handful of 
local district administrators and school principals believe that the State is somehow failing 
to educate its youth. 

This perception of a good public school system with room to improve is shown in 
other results, as well. As Figure 2 shows, about two-thirds of local district administrators 
and school principals think that Oregon’s public schools are in pretry good shape to 
provide the kind of education students will need five to 1 0 years from now. but that some 
work remains to be done. 



Figure 2 


Overall Evaluation of Oregon’s Public Schools’ Ability To Meet the Educational 
Challenges of the Future * 




Local District 
Administrators 

% 


School 

Principals 

% 


ESD 

Administrators 

% 


In good shape, ready to meet challenges 


6 


8 


3 


In pretty good shape but some work needed 


69 


65 


61 


Not in very good shape, a lot of work to do 


20 


20 


32 


In trouble and need to seriously rethink 


3 


4 


4 



* Note : Not all responses are shown. 



Yet, these same local public school officials are less sanguine about the trend in the 
quality of public education in the state and the ability of the public schools to provide a 2 1 st 
Century education. Although school principals are twice as likely to think that the quality 
of public education in the state is improving (52%) as slipping (26%), the margin is a 
relatively narrow 41 percent to 33 percent among local district administrators and 39 
percent to 21 percent among ESD administrators. (See Figure 3.) 
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Figure 3 



Evaluation Of Oregon Public Education Over 
The Past Five Years 
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In other words, one in four school principals and one in three local district 
administrators express the view that public education in Oregon is not improving, and 
indeed, is slipping backward. This includes a 45 percent plurality of local district 
administrators in the Portland metropolitan area, in contrast to a 5 1 percent majority of 
local district administrators in rural Oregon, that is, outside the Portland metropolitan area 
and Willamette Valley, who hold the opposite view. 

Not only do many of the local district administrators express some concern about 
the trend in public education, but as Figure 3 shows, close to one in four local district 
administrators and school principals feel that the state’s public schools are not really in 
very good position to provide students with a 21 st century education and that considerable 
work needs to be done. Some even believe that the schools are in trouble and need 
seriously to rethink how they are educating students. This latter view is most likely to be 
held by local district administrators in districts with fewer than 10,000 students and school 
principals in districts with fewer than 4,000 students. 

What nearlv all administrators and principals agree on is that there is a crisis in 
public education funding in Oregon. Rather than perceiving this as a problem that does not 
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need much attention at this time, more than 90 percent of these local public school 
officials agree that there is a crisis in public education funding in the state that must be 
dealt with quickly. One could choose to discount these results as the standard view of all 
local school officials, but the high degree of consensus and the choice of the term “crisis” 
over “problem” suggests that local school officials are genuinely and seriously concerned 
about funding and its implications for the future. Indeed, a 56 percent majority of local 
district administrators and 55 percent of school principals volunteer funding as one of their 
biggest concerns about the ability of Oregon’s public schools to provide a quality 
education for students in the 21 st century. 

The Oregon Department of Education 

There is no consensus among local district administrators and school principals that 
the Oregon Department of Education plays a central role in helping them provide quality 
education to the students in the districts and in their local schools. (See Figure 4.) 

Figure 4 



Importance of Oregon Department of Education 
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School Principals 



Fairly 
Important 
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Not 
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While 41 percent of local district administrators and 47 percent of school principals 
say that ODE does play an essential or very important role in helping provide quality 
education, just under one-third of each group say it is fairly important, and one in four local 
district administrators and one in five school principals regard its role as just somewhat or 
not that important. Clearly, these local public school officials are not dismissing ODE, but 
neither are the majority prepared to assign ODE a crucial role in providing quality 
education at the local level. We note, however, that nearly half the local district 
administrators in the Portland metropolitan area and in districts with 10,000 or more 
students — more than any other administrators — feel that ODE does play an essential or 
very important role in helping their districts provide quality education. Nearly four in five 
ESD administrators feel the same way, but outside the Portland metropolitan area, the 
figure slips to 37 percent among local district administrators. 

Local public school district administrators rate the performance of the Oregon 
Department of Education about the same way as they rate the quality of public education in 
the state: good, but not top-flight. As Figure 5 shows, few local district administrators or 
school principals are willing to say that ODE is doing an excellent job; more than anything 
else, they believe that it is doing a good job. 

Figure 5 



Job Performance of Oregon Department of Education 
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Indeed, ODE’s reputation among school principals is very solid, with virtually two- 
thirds rating its performance as excellent or good. Among local district administrators, the 
results are very mixed: 51 percent excellent or good, 45 percent fair or poor. There is little 
difference in the ratings by geography or size of district, but ODE’s higher marks among 
school principals come almost entirely from highly favorable ratings among principals in 
those districts with more than 4,000 students and from principals in the Portland and 
Willamette Valley areas. 

Local district administrators, ESD administrators, and school principals in the 
survey also were asked to volunteer what they believe to be the major strengths and 
weaknesses of the Oregon Department of Education. On the positive side of the ledger, 
respondents in all groups are most likely to say that ODE has good people who are 
personable, helpful, and dedicated. Other favorable comments most often provided by the 
local district administrators include: serving as a clearinghouse for information 
(volunteered by 13%), setting common standards and goals (12%), establishing high 
standards (1 1%), providing good educational leadership (11%), providing assistance on 
financial issues (1 1%), developing better curricula (7%), and providing guidance on rules 
and regulations (6%). 9 

On the other side of the ledger, the most common criticism is ODE’s lack of 
personnel, understaffing, and inability to retain a high quality staff (volunteered by 31% of 
all local district administrators, though by just 10% of school principals). At the same 
time, one in five local district administrators and nearly as many school principals also cite 
a lack of adequate funding and resources for ODE. Other weaknesses perceived in ODE 
among local district administrators include: poor leadership (13%), a difficult political 
climate in which to work (9%), inexperience and lack of expertise on the part of some ODE 
perso nn el (9%), late information and slow response time (6%), and a loss of touch with 
reality and too much focus on the peripheral instead of the central issues (5%). 

To explore further the images of ODE held by local district and ESD administrators 
and school principals, each respondent was read a list of 10 qualities and asked to rate the 

9 Note that these percentages come from volunteered responses in which the respondents answered in their 
own words and their own way, not from their selection among a limited and specified set of choices. 
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Oregon Department of Education on each using a five-point scale on which a “5” means 
the quality describes the ODE very well and a “1” means it does not describe the ODE well 
at all. Figures 6 and 7 display these results. Figure 7 summarizes the results among the 
local district administrators by showing a simple arithmetic expression called the 
“differential,” which is nothing more than the proportion of respondents who say each 
quality describes ODE (scores of “4” or “5”) minus the percentage who say the quality 
does not describe ODE (scores of “2” or “1”). For example, the differential of +53 percent 
on “bureaucratic” is the proportion of “4” and “5” scores (61%) minus the proportion of 
“2” and “1” scores (8%), while the +51 percent differential for “competent and 
professional” is 62 percent (the proportion of “4” and “5” scores) minus 1 1 percent (the 
proportion of “2” and “1” scores). 
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Figure 6 



Differential: Qualities that describe the Oregon Department of Education (4+5) Minus Qualities 
Do not Describe ODE (1+2) 
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Figure 7 



ODE’s Image Among Local District Administrators 
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In a nutshell, school principals, ESD administrators, and local district 
administrators from districts large and small all pretty much agree: ODE is competent and 
professional, but also bureaucratic. 

The bureaucratic element shows up in other results as well, as the Oregon 
Department of Education receives mixed reviews on communication. A 43 percent 
plurality of local district administrators indicate ODE is “open and accessible, listens to 
people in the local districts,” but one in five (27%) say this is not the case. ESD 
administrators basically agree with this assessment, while a majority of school principals 
(51%) give ODE favorable marks on this quality. The scores are completely mixed in 
terms of communication from the Oregon Department of Education to the local public 
school officials. As many local district administrators believe that ODE is “informative, 
communicates well” (35%) as believe it is not (35%). Local district administrators in the 
larger districts (over 4,000 students) are especially critical of ODE in this regard, as just 
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33 percent rate ODE highly for being informative and communicating well, while a 
41tpercent plurality rate ODE in the lower portion of the scale on this quality. 

The problem is neither excessive nor insufficient requests for information from the 
local school districts. Indeed, more than 60 percent of ESD administrators, school 
principals, and local district administrators in districts with more than 4,000 students feel 
that ODE requests about the right amount of information. Only among administrators in 
the smaller districts (under 4,000 students) is there a minority of significant size (31%) 
who find the requests for information excessive and time-consuming. 

From the local perspective, the problem lies more in the Oregon Department of 
Education’s use of the information it collects than in its volume. Indeed, less than half of 
local district administrators, ESD administrators, and school principals feel that ODE 
makes adequate use of what it collects. As Figure 8 shows, ODE’s reports, publications, 
and other forms of communication receive solid marks for being “accurate and 
informative” and acceptable marks for being “clear and well-written.” 

Figure 8 



Proportions Who Say Each of Four Qualities Characterize ODE 
Communications all or Most of the Time 
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On the other hand, the scores are lower for being “timely” and “relevant and 
helpful.” In each of these cases, less than half the local district administrators believe these 
qualities describe the ODE’s communications at least “most of the time,” and the school 
principals do not disagree. Note that the figure shows much higher marks on all four 
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qualities among ESD administrators, which indicates that ODE may be communicating 
better with regional than with local district administrators. 

The Oregon Department of Education also receives unimpressive scores on several 
other qualities. Only a minority of local district administrators think of the ODE as 
“creative, innovative” (23%), “flexible, able to adapt to new roles and functions” (29%), or 
“effective, gets things done” (29%). In each case, about one in three say the quality does 
not describe ODE. ESD administrators and school principals are less critical on these 
qualities, but barely more than one in three in either of these latter groups is willing to say 
that either quality describes ODE. Either a plurality or majority takes the middle ground (a 
rating of “3”). 

The results are somewhat more favorable than unfavorable when it comes to 
“leadership, promotes public education throughout the state,” but they hardly constitute a 
rousing endorsement. By 42 percent to 30 percent, local district administrators say the 
Oregon Department of Education provides education leadership, and school principals 
agree by 48 percent to 24 percent. The ESD administrators demonstrate a greater degree of 
consensus than do either of these two other groups, 64 percent to 7 percent, but they are the 
only ones among whom a majority regard the ODE as exerting educational leadership in 
the state. 

Finally, but very important, a plurality of respondents reject the notion that the 
ODE is “intrusive” or “marginal, has little impact on the quality of education.” Local 
district administrators from the smallest and the more rural areas of the state, however, are 
the ones most likely to regard ODE as marginal, including 38 percent from districts with 
fewer than 2,000 students and 31 percent outside the Portland metro and Willamette Valley 
regions. 
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The Role of the Oregon Department of Education: 
Coach, Referee, or Spectator 



To capture a sense of what kind of role local public school officials want ODE to 
play in the educational process, respondents were asked which of the following three 
possibilities best describes the Oregon Department of Education: more of a referee, more 
of a coach, or more of a spectator. Respondents were then asked which best describes the 
role they think ODE should be performing. Figure 9 summarizes the results for local 
district administrators. 

Figure 9 
ODE’s Role 



As Local District Administrators Perceive It/Prefer It 
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The role that local public school officials overwhelming agree that ODE should be 
playing is that of a coach, “helping local school districts provide quality education,” rather 
than performing the role of a referee, “enforcing state and federal regulations in public 
education,” much less sitting on the sidelines and watching the game go by. 
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However, a 46 percent plurality of local district administrators and an equal 
proportion of ESD administrators think that ODE is playing the wrong role: too much of 
the referee, and not enough of the coach. Indeed, just 39 percent of the local district 
administrators who want ODE to play the role of coach believe that it is actually playing 
that role, while the figure increases to 52 percent among both ESD administrators and 
school principals. 

Further analysis suggests two important points. First, as Figure 10 shows, ODE 
may be playing the role local district administrators want in the state’s urban districts more 
than in its “outstate,” rural districts. Local district administrators in the Portland 
metropolitan area are the ones most likely to think of ODE as a coach, while those outside 
the Portland metropolitan and Willamette Valley areas are least likely to see the Oregon 
Department of Education as “helping local school districts provide quality education.” 



Figure 10 

Local District Administrators Who Perceive ODE As Coach 




All local Portland Williamette Rest of State 
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Indeed, a 54 percent majority of administrators in districts with fewer than 4,000 
students and 52 percent in districts outside the Portland metro and Willamette Valley areas 
want ODE to play the role of coach, but perceive ODE as more of a referee than a coach. 

Second, one of the implicit criteria local public school officials are using to judge 
the value of ODE is their perception of the role it plays in the educational process. Those 
local district administrators who feel that ODE is playing the role of coach rather than 
referee or spectator are much more likely to rate the performance of ODE as excellent or 
good (70% coach, 37% referee or spectator). The same pattern holds for school principals. 
Those principals who think ODE is performing a coaching role are far more likely to rate 
the performance of ODE as excellent or good (84%) than are those who think of ODE as 
more of a referee or spectator (47%). Among those who see ODE as more of a referee than 
a coach that helps local school districts, at least 50 percent of ESD administrators, local 
district administrators, and school principals rate ODE’s performance as fair or poor. 
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What is it, then, that local public school district officials say that the Oregon 
Department of Education does best, and in what ways do they feel it may be falling short? 
Respondents were given a list of eight functions, and for each, asked whether it seems to be 
a major priority for ODE, a minor priority, or not something ODE does. Respondents then 
were asked whether the Oregon Department of Education carries out each function very 
well, fairly well, just okay, or not very well for each function the respondents believe to be 
either a major or minor ODE priority. Figure 1 1 summarizes the results. 



Figure 11 
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*Note: Not all responses are shown 

As these results amply demonstrate, at least 70 percent of local district 
administrators report that ODE concentrates its efforts on at least five priorities: 
monitoring and enforcing state and federal regulations, assessing student progress toward 



statewide goals, lobbying the state legislature, and providing leadership in curriculum and 
instruction. The reason ODE does not get higher scores for playing the role of coach also 
is evident in Figure 1 1 . Half or fewer local district administrators feel that ODE assigns 
high priority to those functions most clearly and directly related to helping local school 
districts provide quality education: organizing and providing staff development 
opportunities, helping develop local district capacity for improving education, and assisting 
local school districts with financial management. 

Figure 1 1 also makes clear what local district administrators believe ODE does 
best: managing and monitoring federal regulations, enforcing state regulations, and 
assessing student progress — all of which sound more like referee than coaching activities. 
Note, however, that there is no function that more than 27 percent of local district 
administrators say ODE does not do very well. 

Local district administrator ratings of the performance of ODE on each of these 
eight functions are presented graphically in Figure 12. 
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Figure 12 



Positive Assessment Of ODE 
On Selected Efforts 

(% local district administrators rating ODE as doing each veryffairty well) 



Manage fed. programs 




Enforce state regs 


1 


Assess students 


■ i'~ 


Leader on curriculum 


| 52% 


Lobby legislature 


I 4 ** 


Financial mgmt assist 


| 34% 


Dev. opptys/programs 


| 33% 


Help local improve ed. 


y 1 31% 



Further analysis reveals some important differences of opinion among local district 
administrators from different kinds of districts around the state: 

• Administrators from districts with fewer than 2,000 students think that ODE 
does more in the way of organizing and providing staff development 
opportunities. Nearly half (47%) say this is a major priority for ODE, and 
53 percent say ODE does a very or fairly good job of it. 

• Administrators from these same smallest districts also rate ODE higher than 
do other administrators when it comes to representing public education in 
the state legislature. Indeed, a 56 percent majority say ODE does very or 
fairly well in this regard. 

• Administrators from the largest districts (10,000 or more students) and 
smallest districts (fewer than 2,000) rate ODE higher than do other 
administrators when it comes to helping develop local district capacity for 
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improving public education. Majorities of administrators in the largest and 
smallest districts believe ODE makes this issue a major priority. In the 
smallest districts, however, 42 percent also say that ODE is doing very or 
fairly well in this respect; among local district administrators in districts 
with 10,000 or more students, this figure drops by more than half to just 20 
percent. 

Education Service District administrators hold many of the same views as the local 
district administrators, but several differences are well worth noting. 

• ESD administrators are much more likely than are local district 
administrators to hold ODE’s legislative lobbying efforts in high regard. 

The proportion rating ODE’s efforts very or fairly good rises from 43 
percent among local district administrators to 79 percent among the ESD 
administrators. 

• ESD administrators also give ODE more very or fairly good ratings (75%, 
compared to 61% among local district administrators) on assessing student 
progress toward statewide goals. 

• On the other hand, ESD administrators give ODE somewhat lower marks 
for leadership in curriculum and instruction. The 52 percent majority of 
local district administrators who say ODE is doing a very or fairly good job 
in this regard drops to 39 percent among ESD administrators. 

School Improvement Visits 

Because it is one of the Oregon Department of Education’s highest priorities, a 
source of major expenditures, and offers the most direct contacts with local public school 
districts, a series of questions was included in this survey to understand the value of school 
improvement visits from the perspective of the local district administrators and the 
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principals in the schools themselves. Three in five local district administrators say they 
have received a school improvement visit from ODE within the past three years, including 
35 percent who recall a visit in the past year alone. Another 21 percent report a visit within 
the past four or five years, and 14 percent more than five years ago. 

As Figure 13 shows, more than 70 percent of local district administrators (and not 
shown in the figure, more than 80 percent of school principals) say the visits do more good 
than harm. A 51 percent majority of school principals and a 41 percent plurality of local 
district administrators, however, regard the visits as only somewhat worthwhile, rather than 
extremely valuable (6% of administrators, 3% of principals) or quite helpful (24% and 
27%, respectively). This may not be “damning with faint praise,” but neither do these 
results constitute a ringing endorsement. 



Figure 13 



Local District Administrators’ View of School Improvement Visits * 
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* Note: Lefthand graphs are responses to Question 10B of the survey: “How useful are [School Improvement 
Visits} ~ extremely valuable, quite helpful, somewhat worthwhile, or more harmful than helpful? " Right 
hand graphs display responses to Question 11 A: “Compared to other things that the Oregon Department of 
Education could do to help, would you say that these site visits are the most helpful thing, among the most 
helpful, somewhat further down the list, or not that valuable? ” 

When those who say they have had a visit recently enough to offer an opinion are 
asked to talk about the benefits and problems of school improvement visits, 70 percent of 
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local district administrators and more than three-fourths of school principals volunteer at 
least one benefit, but 62 percent of local district administrators and 56 percent of school 
principals also volunteer at least one problem with the visits. 

On the benefits side, local district administrators talk most about compliance — 
about discussing standards and the extent to which their students are meeting state 
standards, and about laws and regulations and making sure the local schools are meeting 
and complying with state and federal regulations. School principals are most likely to say 
the visits help them in terms of perspective — evaluating what they are and ought to be 
doing, helping them see things from a different point of view, and increasing their 
awareness of issues. The topics discussed with the second-most frequency: local district 
administrators talk about perspective, while principals talk about compliance. 

The central question is the value of school improvement visits compared to other 
ODE activities, and from the local perspective, the visits just do not measure up. In fact, 
just 23 percent of local district administrators and 24 percent of school principals regard 
school improvement visits as either “the most helpful” or “among the most helpful” things 
ODE could do to help them in their district. A 47 percent plurality of principals and a 38 
percent plurality of local district administrators rate school visits as “somewhat farther 
down the list.” Indeed, one in four local district administrators either say these visits are 
“not that valuable” (10%) or actually do more harm than good (16%). The comparable 
figures among school principals are 9 percent and 6 percent. 

Figure 14 uses the results from a somewhat different question to show this rather 
tepid assessment of the school improvement visits among local district administrators, 
especially among the larger (enrollment of 10,000 or more) and Willamette Valley 
districts. Local district administrators demonstrate more enthusiasm the more recent the 
visit, but “extremely valuable ” and “quite helpful” evaluations still do not top two in five 
even if the site visit has occurred within the past year. 
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Figure 14 



Local District Administrators’ View of School Improvement Visits Revisited * 
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* Note: Lefthand bar graphs refer to survey Question JOB: “How useful are [School Improvement Visits] — 
extremely valuable, quite helpjul, somewhat worthwhile, or more harmful than helpful? ” Righthand bar 
graphs again refer to Question 1 OB, but with an indication of the recency of the visit. 

Perhaps most telling, even among those local district administrators who think there 
is at least some value in the school improvement visits, only 17 percent would be willing to 
commit more of their own resources to continue the visitation program, while three-fourths 
say they no longer would have site visits done. Principals are even less enthusiastic: just 8 
percent would be willing to commit more of their own resources, while five in six would 
skip the site visits. 

Finally, just 15 percent of all local district administrators believe that school 
improvement visits should be conducted solely by staff members from the Oregon 
Department of Education. More than one-third (35%) think that these site visits would be 
more valuable if they were conducted by educators from other districts, and nearly half 
(47%) would prefer a combination. Once again, the results among school principals differ 
little from those of local district administrators. 



The Oregon Educational Act For the 21 st Century 

One of the biggest challenges for the Oregon Department of Education is to help the 
State's public school system prepare for and begin to implement the Oregon Educational 
Act for the 21 st Century. The vast majority of local school officials feel that they 
understand the Act, including 75 percent of ESD administrators who say they understand it 
very or fairly well, 82% of local district administrators in districts with fewer than 4,000 
students, 89 percent of those with more than 4,000 students, and 93 percent of school 
principals. 

From the perspective of ESD administrators, local district administrators, and 
school principals, ODE has not allowed this responsibility to languish but at the same time, 
they feel there is much yet to be done. As Figure 1 5 demonstrates, both local district 
administrators and school principals are about evenly split on whether or not ODE is 
making significant effort or some progress. Few either believe that great strides have been 
made or that ODE has not yet begun to meet the new demands. 

Figure 15 

ODE Progress on Oregon Educational Act for 21 SI Century 
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It is, however, evident that local public school officials believe the pace of progress 
has been uneven and that ODE has accomplished much more in some areas of change than 
in others. Six areas were tested, with the results shown in Figure 16. 



Figure 16 

ODE Progress toward Implementing the Oregon 
Educational act for the 21 st Century 
(Has Already Accomplished or Is Well on its Way) 
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These same results are shown visually for local district administrators in Figure 17. 



Figure 17 
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The consensus of opinion at the local level is that ODE has done more to define and 
articulate the education mission of the State’s public school system than to communicate 
the vision clearly and coherently, and that ODE has accomplished more in terms of 
establishing performance standards for students than in collecting the necessary data to 
measure students’ progress toward these goals. 

It is when implementation of the reform act takes the form of the most direct 
assistance to schools and school districts that the local public school officials see the least 
progress. Just one in five ESD administrators and one in five local district administrators 
believe that ODE is well on its way to providing curriculum development, technical 
assistance, and professional development to local schools so they can meet the performance 
standards that are being set for their students, and to serving as a clearinghouse for the 
exchange of successful ways to improve education and student performance. School 
principals are a bit more optimistic, but just one in four thinks that ODE is well on its way 
to providing curricular, technical, and professional development assistance, and just one in 
three feel this way when it comes to serving as a clearinghouse for successful educational 
reforms. 

In other words, local public school officials see ODE as having made significant 
progress in defining the education mission of the public schools and what that means in 
terms of student performance. However, ODE has accomplished less, so far, in terms of 
articulating and communicating that mission and collecting the information to measure 
student progress toward the goals imbedded in the education mission; and it is either just 
beginning or has not yet begun to develop the clearinghouse and local assistance programs 
that would develop the local capacity to achieve the education mission of the State’s public 
school system. 
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Final Advice From Public School Officials 



To provide as much guidance as possible to the Oregon Department of Education, 
school principals, local district administrators, and ESD administrators were asked at the 
conclusion of each of their interviews to volunteer what advice they would give to ODE so 
it can better serve the needs of their own districts and schools. 

Several points clearly emerge from the pattern of responses. 



• The number-one desire is for ODE to secure more funds, so it can hire more 
people, keep its better people, and experience less turnover. 

• Second is for ODE to take more time to understand the issues, problems, 
and needs in the local districts and schools — to listen better, be more 
receptive, and not treat local school officials “like the enemy.” 

• Third, ODE should provide more and better educational leadership, not only 
with the public statewide, but within the political arena of state government. 

• Fourth, ODE needs to be more focused and to “stay the course,” avoid 
inconsistencies, and to focus more clearly and sharply on the central issue of 
educating children. 

• Finally, ODE must provide better communication, especially better, more 
relevant, and more easily understood information, to the local schools and 
districts. 



This is neither a harsh list of suggestions containing implicitly severe criticisms of 
ODE, nor is it an impossible list to address. If anything, this agenda indicates a clear 
desire to support the Oregon Department of Education and to work more closely with it in 
the process of improving the quality of teaching and learning in Oregon’s public schools. 
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Chapter 6 



PERSPECTIVES ON IMPLEMENTATION 



Creating a high performance education system is a daunting challenge, one 
that state legislatures are addressing with policies to significantly raise academic 
standards even while extending to local districts and schools unprecedented 
flexibility in meeting jointly agreed-on performance benchmarks. Oregon was one 
of the first states to adopt the standards/ flexibility approach to school reform but 
it is certainly no longer alone in this endeavor. Forty-eight states have committed 
to developing state level academic standards of one kind or another; 15 of these 
have already promulgated standards in the core subject areas that are clear and 
well-grounded. A total of 46 states require schools to report data on student 
performance. 

A major lesson from other states' experiences with standards-based reform 
is that change cannot be accomplished in piecemeal fashion. Higher standards 
and assessments will not improve student performance by themselves unless 
schools both are granted the flexibility to seek site-specific solutions and helped to 
gain the capacity to upgrade instruction. Neither reform rigidly prescribed by 
statewide mandate nor reform as interpreted by completely autonomous 
community schools has an inspiring track record in improving student 
performance. What seems to offer the best prospect of success is a simultaneous 
"top down" and "bottom up" strategy of change, where the multi-faceted 
leadership of the state, as exercised through its state department of education and 
including an implementation strategy of investment, information and assistance, 
complements local initiatives in fostering high academic performance. 

The basic perspective to keep in mind, however, is that standards-based 
reform is still a work in progress, not just in Oregon but throughout the nation. It 
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cannot yet be said that any state has accomplished the difficult, delicate and 
ambitious task of reinventing its public school system so that a much higher 
proportion of students is being well prepared to meet the challenges of the 21 st 
century. Nevertheless, the experiences of other states offer valuable lessons — of 
both a hopeful and monitory nature— for how the Oregon Department of 
Education might go about leading the State's standards-based reform initiative. 
Accordingly, this chapter describes significant aspects of the school reform 
experience of three states— Kentucky, California and Rhode Island. 

KENTUCKY: THE POWER AND PERILS OF 
HIGH STAKES ASSESSMENT 

In June 1990, the Kentucky legislature passed the Kentucky Education 
Reform Act (KERA), a sweeping restructuring of the state's public school system. 
While the timing of Kentucky's law coincided with Oregon's reform legislation, 
the impulse for change was quite different. KERA was compelled by a decision 
handed down by Kentucky's Supreme Court declaring the state's "entire system 
of common schools" to be unconstitutional. This unprecedented zeroing out of 
the status quo was based not just on documented inequities in the school funding 
system but on palpable dysfunction in the schools themselves. Kentucky 
traditionally had paid little for education and— with an adult literacy rate of 50 th 
in the nation, a high school completion rate of 50 th in the nation and endemic 
poverty —had gotten little out of it. 

KERA was the aggressive, standards-based plan to change all that. Its 
essential architecture— higher educational standards and new statewide 
assessments to be accompanied by rewards and sanctions according to school 
performance— was supplemented by a host of support systems intended to 
address a spectrum of anticipated challenges. Thus, KERA made staff 
development a line item in the state budget, increasing money allocated to helping 
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teachers improve their instruction 23-fold and creating eight regional service 
centers to provide training and technical assistance to school districts. The law 
instituted preschool programs for at-risk four-year-olds and children with 
disabilities (now serving 82 percent of eligible children); an after-school and 
summer school program for students who need more time to learn; and family 
resource centers to put impoverished students and families in touch with needed 
health and social services. To guarantee local involvement in the change process, 
the state also created School-Based Decision Making Councils consisting of 
teachers and parents. These councils were invested with real authority (for 
example, the SBDM Councils participate in hiring school principals). 

The Kentucky Department of Education was abolished and reorganized in 
June 1991. The reorganization created divisions basically corresponding to the 
various functions outlined in the new law. Five major studies of the new, 
reconstituted KDE offer a mixed picture of how it has promoted change. Its 
clients— school and district administrators — still regard KDE as heavily 
bureaucratic and disposed to control the direction of reform through top-down 
fiat. They fault KDE for being inconsistent and confusing in the advice it gives but 
praise the work ethic of its personnel. The most substantive criticism of KDE's 
performance, however, has revolved around one key element of the reform 
program: the high-stakes assessments. 

Of all KERA's initiatives, the area of assessment and accountability has 
certainly proved the most controversial. The assessment system, known as the 
Kentucky Instructional Results Information System, or KIRIS, consists of a variety 
of formats, including writing and math portfolios of student work, open-ended 
and multiple choice questions, essay examinations and performance events. It is 
administered to all students in grades 4, 8, and 11/12 in all schools in Kentucky. 
KIRIS is a "high stakes" testing system because important consequences flow from 
how students perform on it on a schoolwide basis. Based on comparison between 
data collected at the beginning of an accountability cycle and average scores over 
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the next three years, individual schools are placed into one of five categories: a) 
eligible for monetary rewards, b) successful, c) not meeting threshold or 
improving, d) in decline, and e) in crisis. Educators in the first category received 
approximately $26 million in bonus money in 1995. Certified staff in schools in 
the lowest category, "in crisis," have not yet suffered the extreme sanction called 
for in the original law— being placed on probation. However, schools so 
designated must develop a school improvement plan and, in many cases, are 
assigned a "Kentucky distinguished educator" to advise school personnel on 
implementing the plan. 

The problem with high stakes assessment, it turns out, is that the 
assessment instruments themselves are sufficiently novel that they lack a high 
degree of reliability, all of which makes the assignment of rewards and 
punishments unacceptably arbitrary. For example, the KIRIS accountability index 
of schools purported to show a 22 percent increase in performance at grade 4, 13 
percent increase in grade 8, and 9 percent increase at grade 12, in the first four 
years of the reform law. But external research evaluations called into question the 
accuracy of these results. For one thing, KIRIS results did not correlate very 
closely with national standardized tests such as the Comprehensive Test of Basic 
Skills or National Assessment of Educational Progress, as would be expected. An 
evaluation by the Office of Education Accountability, a legislative agency charged 
with monitoring KERA activities, found the KIRIS scores to be "substantially 
inflated" and the technical test procedures used to derive them seriously flawed. 

The Kentucky Department of Education has taken steps to repair the 
assessment problems, including (at the explicit direction of the legislature) 
temporarily halting the use of math portfolios as a basis for evaluating student 
achievement. Still, the damage lingers. A 1995 survey showed that only about 
one in four Kentucky teachers believes KIRIS to be a very good or moderately 
good measure of how effectively schools are performing. 
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There are important lessons for other State departments of education in 
Kentucky's experience. First, the evidence confirms that statewide assessment has 
proved to be a powerful lever in influencing what goes on in the state's 
classrooms. Researchers have found positive effects on instruction, especially 
increases in student writing. On the other hand, the assessment miscue underlines 
the fact that SDEs in standards-based reform states are basically in the position of 
a pilot learning how to fly the plane on the way to the target. State departments of 
education are expected to provide leadership, but the terrain itself is uncharted. 

In such a situation, there is a high likelihood that mistakes will be made. All the 
players in education reform need to recognize that. They should also realize, as 
one informant told us, that there may well be the greatest advantage to students 
and society in pressing ahead and muddling through, as Kentucky has done, 
correcting problems as they crop up, rather than waiting until the perfect solution 
has been invented (which day will never come, given the complexity of the task). 

CALIFORNIA: PROFESSIONAL DEVELOPMENT THROUGH 
THE SUBJECT MATTER PROJECTS 

During the decade of the 1980s, California initiated an ambitious reform of 
K-12 education. The predominant characteristic of that effort was its systemic, 
well articulated nature. The key elements in the effort to improve teaching and 
learning were curriculum frameworks, which attempted to define a professional 
consensus concerning what students should learn in specific subject areas; 
statewide assessments based on the frameworks; and professional development 
opportunities and the selection of instructional materials also tied to the 
frameworks. The Superintendent of Public Instruction and State Department of 
Education played prominent leadership roles in putting all these elements in 
place. For the purposes of this discussion, we will focus on one aspect of the 
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systemic reform: SDE leadership in greatly enhancing professional development 
opportunities. 

Virtually all educators are doing the best job they can; if they knew how to 
produce the desired outcomes stipulated in statewide standards, they would 
already be doing so and reform would be a moot topic. The difficulty is not one of 
motivation or willingness but rather of capacity. MAP knows of only two ways to 
improve instruction: either hire staff who already possess the desired ability or 
improve the skills and knowledge of the incumbent cohort of teachers. Assuming 
the latter is the only feasible short-term solution, MAP offers the California Subject 
Matter Projects (CSMP) as an example of one effective way California addressed 
professional development of teachers. 

What are the CSMP? They are a permanent, state-funded, statewide 
network of discipline-based entities which provide professional development for 
K-12 teachers. There are projects in nine core subjects (Arts, Foreign Languages, 
History-Social Sciences, International Studies, Mathematics, Physical Education 
and Health, Reading and Literature, Science, and Writing) at 97 sites throughout 
California. Each year approximately 20 percent of California's teachers are served 
by the CSMP. In stark contrast to what usually passes for professional 
development in education, the CSMP bring about meaningful changes in teachers' 
classroom performance by working with teachers over an extended period. The 
core of the learning experience is a 3-4 week intensive summer institute 
augmented during the school year and in subsequent school years by a series of 
follow-up sessions and workshops. 

Outside evaluations of the CSMP have been uniformly positive and 
consistently identify a lasting formative effect on instructional practice inside the 
classroom 10 . (To be sure, the effectiveness of individual projects has varied from 
time to time and discipline to discipline.) The Projects have also been models of 



10 See, for example, Mark St. John: “The California Subject Matter Projects: A Summary of Evaluation 
Findings”, 1993-1996, Inverness Research Associates, December 1996. 
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cost effectiveness derived from economies of scale only a statewide program can 
enjoy. CSMP's average annual cost of $305 per participating teacher compares 
favorably with $612 for California Eisenhower Projects, $3000 for National Science 
Foundation LSC Projects and $4000 for the California Mentor Teacher Program. 

CSMP rely on interagency cooperation but do not require the day-to-day 
management of the State Department of Education The budget and 
administration of CSMP reside in the Office of the President of the University of 
California. More than half of the sites are located on California State University 
campuses. Nearly half of the directors are classroom teachers and the individual 
projects report extensive working relationships with county offices of education, 
local school districts, curriculum associations and various regional consortia. 
Although the CSMP were conceived by the Superintendent of Public Instruction 
as an integral part of the reform strategy, the SDE plays only an advisory role in 
their operation. 

There are two lessons for SDEs from the California Subject Matter Projects 
experience. First, the CSMP are a good example of an SDE demonstrating 
leadership in an arena that was not specifically authorized by law. From the mid- 
1970s California's SDE dedicated a portion of its discretionary federal funds to the 
support of the Bay Area Writing Project. In the early 1980s, responding to the 
need to enhance the capacity of teachers, the SDE diverted discretionary funds 
from lower priority activities to establish what were then called Curriculum 
Implementation Centers in each of the disciplines following the Writing Project 
model. These were housed in county offices of education. The existence of the 
CIC helped build a constituency for legislation in 1988 creating the CSMP and 
funding them in the state budget. 

The second lesson is that the best expenditure of resources is the one that 
has a large multiplier effect in terms of influencing what happens in the classroom. 
The current Oregon Department of Education strategy of funding individual 
teachers to engage in "action research" in their individual classrooms may be a 
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good use of funds for the individual classroom but it fails the test of generating 
leverage for system-wide improvement. Sometime SDEs need to stop doing good 
things in order to free up resources to pursue the most powerful strategies for 
improving teaching and learning at the classroom level. 

RHODE ISLAND: REORGANIZING TO EMPHASIZE TEACHING 

AND LEARNING 

Peter McWalters was named Commissioner of Education for the State of 
Rhode Island in 1992. He assumed leadership of a very traditional SDE which saw 
as its mission regulating and enforcing state and federal program rules. Rhode 
Island's response to "A Nation at Risk" was the "Basic Education Plan" — a four- 
inch-thick handbook of inputs (required minutes of math instruction, bathrooms 
per students, etc.) that were mandated for local school districts. Not surprisingly, 
districts viewed the Rhode Island Department of Education (RIDE) as the 
education police, helpful only in the context of interpreting and enforcing 
regulations. 

Although initiated primarily by the Regents (equivalent to Oregon's State 
Board of Education), school reform in Rhode Island exhibits many characteristics 
similar to Oregon's strategy. Both set learning and performance goals for 
students; both provide for statewide assessment balanced by local autonomy for 
deciding how to achieve goals; both call for collaboration between schools and 
other social agencies; and both place the SDE in a leadership role. 

Commissioner McWalters' main goal for RIDE at the outset was to 
transform it from being the enforcer of rules to the leader of education reform. He 
faced familiar obstacles in doing so: a strong tradition of local control; a history of 
budget reductions; half of RIDE's budget from federal sources which apparently 
constrained how it could be used; and employees represented by an active union. 
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None of these conditions has changed in the intervening years, but RIDE has — 
after a struggle. 

In 1993, the Commissioner reorganized RIDE with primary emphasis on 
reducing reporting levels and flattening the organization. He left the arrangement 
of units, structured by funding source, unchanged. As a result, while there was 
ample evidence of employee support for the reorientation, the desired change of 
behavior did not follow. 11 A second reorganization was initiated. 

The thrust of the second reorganization was to emphasize the importance 
of teaching and learning in the structure. Responsibilities of a single deputy 
commissioner were bifurcated into administrative and education program 
assignments. An assistant superintendent with experience managing complex 
education change was hired (from a local school district) to lead the education 
program functions. The second significant change was to arrange all professional 
employees into cross-functional teams with specific responsibilities for improving 
local district capacity. A third change, which has proved to be somewhat 
controversial, was the implementation of a system of client feedback concerning 
RIDE interactions. 

Some recommended changes have been postponed due to inadequate 
resources, bargaining agent resistance, political sensitivities and so on. 
Nevertheless, the change to a capacity building orientation seems to have firmly 
taken root. This change is sufficiently dramatic that a recent newspaper column 
(The Providence Sunday Toumal, 2/8/ 97) took note of it, reporting 
commendations from district superintendents saluting the new orientation. 



11 Guthrie, James W. et al. “Maximizing Returns on Rhode Island’s Education Investments: A Management 
Consulting Report to the Rhode Island Board of Regents and Rhode Island Department of Education,” 
Management Analysis and Planning Associates (April 14, 1995). 




156 



151 



Chapter 7 

CONCLUDING REMARKS 



This evaluation of the Oregon Department of Education has endeavored to 
assess ODE within the context of that agency's new responsibilities to implement a 
more challenging, and more complex, system of education. MAP has reserved 
this final chapter of the report to underscore a few points that we believe are 
critical to Oregon's continued education progress. 

First, policy makers must constantly keep in mind how difficult systemic 
change is. Education reform is a task often underestimated. But changing an 
institution and a social process— and education is both of these— is among the 
most challenging of undertakings. 

The kind of wholesale education reform in which Oregon is engaged is 
fundamentally about changing habits of mind. It is about shining a light on 
unexamined tradition, scrutinizing habitual ways of doing things, and learning 
new behaviors and approaches. It takes time, training, patience, and a rather high 
tolerance for ambiguity along the way. 

Second, education improvement is a team effort. The governor, legislature, 
ODE, local districts and schools, and Oregon citizens all have important roles to 
play in ensuring that the State's students have the opportunity to meet the 
academic expectations the State has laid before them. 

Third, we emphasize here, as we have throughout the report, that 
education improvement is rarely advantaged by more laws or added regulations. 
As research has amply shown, "You can't mandate what matters most." 12 What 
matters most are capacity and will. These can neither be legislated nor ordered. 
They must be developed and honed over time. 



12 Berman, Paul and Milbrey Wallin McLaughlin. Federal Programs Supporting Educational Change, Santa 
Monica, CA: The Rand Corporation, 1975. 
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Finally we take up a question we know is on the minds of some Oregon 
policy makers. Is there a need for a department of education? MAP's answer is an 
unequivocal "yes." 

Systemic reform has been described as a combination of "top down" and 
"bottom up." In other words, authority and responsibility for, and ideas about, 
education improvement move both from the statehouse to the schoolhouse and 
from the classroom all the way up through the policy system. The role of ODE, 
indeed of any state department of education, is to create and communicate a 
vision of educational excellence, based on the design contained in state policy, and 
then implement a strategic plan to operationalize this vision. ODE additionally 
needs to develop and put in place a system to hold people accountable for 
education results. And it must provide assistance (principally in the form of 
capacity building) to ensure that desired results can be achieved. The Oregon 
Department of Education often serves as the critical two-way link between state 
officials on the one hand and local educators on the other. It has an essential— and 
nontransferable — role and mission. 

MAP has been pleased to have had the opportunity to conduct this 
appraisal of the Oregon Department of Education. We hope that the information 
contained in this study will prove useful to Oregon policy makers, educators and 
citizens. 
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RESPONSE TO REPORT 



OREGON'S DEPARTMENT OF EDUCATION 



Norma Paulus 

STATE SUPERINTENDENT OF PUBLIC INSTRUCTION 



April 18, 1997 



Sam Cochran 

Acting State Auditor 

Division of Audits 

Office of the Secretary of State 

255 Capitol Street, NE 

Salem, Oregon 97310 

Dear Mr. Cochran: 

Enclosed is the Department of Education's response to the Management 
Analysis & Planning Associates' draft program evaluation dated March 24, 1997. 



Sincerely, 




C. Gregory McMurdo 
Deputy Superintendent 
of Public Instruction 



Enclosure 



255 Capitol Street NE, Salem, Oregon 97310-0203 • (503) 378-3569 • Fax (503) 373-7968 



Oregon Department of Education Response to Draft Audit 

April 18, 1997 

Following is the Oregon Department of Education’s response to the report, “TO LEAD 
EDUCATION REFORM, An Appraisal of the Oregon Department of Education” by 
Management Analysis & Planning Associates, dated March 24, 1997 (MAP report). 

1. Findings 

Leadership 

1 . ODE employees, while dedicated and hardworking, do not perceive themselves as 
having an important role in improving teaching and learning. 

The Department agrees that its employees are dedicated and hardworking. It 
disagrees with the statement that they “do not perceive themselves as having an 
important role in improving teaching and learning.” The actions of Department staff 
make clear that they view themselves as responsible for improving student 
achievement. During school improvement visits, Department staff review district 
assessment scores and dropout rates with district staff. Department staff view the state 
assessment as a tool to measure student progress and improve learning. Oregon’s 
school improvement effort is based on the premise that assessment results and student 
progress are tied directly to improved curriculum, instruction and student learning. 
Department staff have far too much experience in this diverse state to suggest there is 
one strategy for change that can be applied across the board to improve student 
achievement. 

As MAP acknowledges, implementing a standards-based system is complex. It must 
be done sequentially. The Department began developing a criterion-referenced 
standards and assessment system in the 1980s. The first statewide assessment for all 
students occurred in 1991. Following 1995 amendments to the Oregon Educational Act 
for the 21 st Century, the K-12 content and performance standards were not ready for 
adoption by the State Board of Education until September and December 1 996. 

2. ODE’s current organizational structure neither conveys nor facilitates an activist 
leadership role in improving teaching and learning. 

The Department disagrees with this finding. Department positions were cut 
substantially due to Ballot Measure 5 in 1991. The organization was flattened to make 
it more responsive to the educational changes called for in the Oregon Educational Act 
for the 21 st Century. 

Department staff bring a depth of knowledge and experience, not to mention 
professional affiliations, to their areas of expertise. Though too lengthy for this 
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document, a few examples may illustrate the point. The Department’s social sciences 
specialist is outgoing president of the Oregon Council for the Social Studies. The 
science specialist (a full-time, not part-time staff member) was president-elect of the 
Oregon Science Teachers Association. The Department's new mathematics specialist 
begins work after his teaching year ends this summer. Despite demanding jobs that 
limit available time, Department staff do attempt to communicate across offices. 
Managers from each office in the Department meet every two weeks. Assessment and 
professional technical education managers and staff meet frequently with curriculum 
staff and compensatory education. 

3. Current conditions preclude ODE from securing as employees the broad range of 
educational experts necessary to carry out the requirements of education reform. 

The Department agrees but questions how it can raise salaries within the state system. 
Statutory and other changes would be necessary and undoubtedly would be 
strenuously resisted by state employee union members. 

4. ODE does not systematically employ comprehensive planning and budgeting as 
management tools. 

The Department disagrees. Each biennium, the Department develops a 
comprehensive plan for implementing the Oregon Educational Act for the 21 st Century 
and other state and federal laws. Staff discuss goals to improve education in the state 
and use the goals to draft legislation. All items with budgetary implications become part 
of the Department’s budget submitted to the Governor and subsequently to the 
legislature. 

5. Few statutory changes are required to enable ODE more effectively to take a 
leadership role in improving teaching and learning in the state. 

The Department agrees that Oregon’s one-volume education code provides a workable 
framework for the state’s educational system unlike highly detailed, prescriptive, multi- 
volume codes often found in other states. The Department disagrees, however, to the 
extent that certain changes recommended in the MAP report require additional state 
funding at a time when state education dollars are limited. 

6. The elected office of superintendent of public instruction tends to fragment 
responsibility for education. 

The Department agrees. State Superintendent of Public Instruction Norma Paulus and 
Department staff have long held that the state superintendent should be appointed by 
the governor, not elected statewide. Change may be possible now as the governor, 
state superintendent and many legislative leaders support it. Several previous efforts to 
make the change failed as voters preferred direct election. 
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Monitor and Enforcement 



7. ODE has undertaken efforts to transfonn itself from a regulatory organization to a 
school assistance organization. However, despite its efforts to move from regulation to 
assistance, ODE continues to conduct most of its work as if it were primarily a 
regulatory and compliance organization. 

The Department agrees that it is becoming a school assistance organization. It 
disagrees with the finding that the Department conducts most of its work as if it were a 
regulatory and compliance organization. Due to limited resources, the Department first 
focused on developing the standards and assessment system and refocusing the 
standardization process. In 1991 and earlier, standardization staff operated separately 
from school assistance staff and placed a major emphasis on compliance. When the 
Department reorganized in 1991, barriers separating offices were dissolved and staff 
began working together on school improvement. The new school improvement visits, 
combining monitoring with technical assistance, were one result. The governor recently 
signed Senate Bill 179, introduced at the request of Superintendent Paulus, to eliminate 
frequent on-site visits to all districts. This law will greatly reduce the amount of time 
specialists spend on school improvement visits, allowing them to focus on building 
capacity to improve teaching and learning in their regions. 

8. ODE has instituted a statewide assessment program but may not be sufficiently 
positioned to take full advantage of its power to improve teaching and learning. 

The Department disagrees with the finding that it is not positioned to take full advantage 
of its assessment system. MAP’s conclusion is premature and demonstrates 
unfamiliarity with the Department’s implementation plan. 

The Department allocates time and funds to methodically and sequentially create the 
curriculum and assessment system. It is important to recognize the steady, persistent 
development and implementation of the system. The academic content standards have 
been developed but teacher resource documents are not yet complete. The state 
assessment system will be completed by 1998-99.' English and mathematics 
assessments are in place. The Department is field testing science assessments and is 
developing assessments in the social sciences (history, civics, geography and 
economics). Development will take time, will involve subject matter associations and 
many other partners as well, and will require funding. For a more complete discussion 



1 Statewide assessments will occur in the following years in the following subjects: 

1996- 97 - English and mathematics 

1997- 98 - English, mathematics and science 

1998- 99 - English, mathematics, science and social sciences (history, geography, civics and economics) 



of the assessment system, see “Review of the Oregon Statewide Assessment Program: 
The Report of the External Review Team." 2 

The assessment system is intended to play a key role in improving education in the 
state. However, it cannot be the sole indicator of a quality school system. Other 
indicators might include high school dropout rates, percentages of college students 
taking remedial courses, percentages of high school graduates attending college, 
percentages of high school graduates working full time and earning more than 100 
percent of the poverty level, adult literacy rates, educational attainment of faculty and 
school climate. Some of these indicators may be difficult to measure. 

Capacity Building 

9. Many key ODE staff members convey an understanding of the importance of 
providing ongoing assistance to districts and schools. However, ODE has no apparent 
comprehensive plan for building local school and district capacity through ODE- 
proffered technical assistance. 

The Department agrees that staff members understand the importance of ongoing 
technical assistance and have a progressive view of professional development. The 
Department disagrees with the finding that it has no plan for building local capacity. 

The Department had to develop its standards and assessment system before it could 
begin providing technical assistance to implement those standards. Just four months 
ago, the State Board of Education adopted academic content standards, performance 
standards in English and mathematics and career-related learning standards. The 
Department is developing a comprehensive implementation plan. Department 
managers and staff meet regularly to flesh out the implementation plan and ensure that 
all office activities support improved teaching and learning. The Department is working 
with an implementation team from the governor’s office, Oregon Business Council, 
community colleges and higher education to describe the next steps to implement 
school improvement goals statewide. 

In June, Department staff will meet with professional organizations to draft professional 
development plans in each academic content area. 3 The plans will be reviewed by 
elementary, middle and high school level professional organizations, Oregon 
Association of Education Service Districts, the Confederation of Oregon School 
Administrators and the Oregon Education Association. 



2 External review team members were Robert Linn, University of Colorado; Joseph Ryan, University of South 
Carolina; Paul Sanifer, American College Testing Program; Edward Roeber, Chair, Council of Chief State School 
Officers. The report was published in October 1992. 

3 The Department committed not to remove teachers from classrooms during the school year so all professional 
development activities will occur outside the regular school day and year. 
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The Consolidated District Improvement Plan connects the state's vision for standards- 
based education with district planning requirements. Since May 1996, Department staff 
have conducted a series of 50 workshops attended by nearly 1,200 educators to help 
them build capacity for local improvement planning. The workshops focused on 
building capacity, not merely producing a written document. One workshop, for 
example, focused on state performance standards, the state assessment system and 
interpreting state test results to inform local goals. 4 Department staff help districts 
connect program and curriculum changes required for the Certificates of Initial and 
Advanced Mastery to district planning. The Department also works with the 
Confederation of School Administrators to support district decision making based on 
analyzing student achievement results. 

The Department supports a variety of other capacity building activities. With the 
Confederation of School Administrators, it funded the Principal Leadership Network. 
Three hundred principals meet in regional study groups to develop regional 
professional development plans to build administrative skills. This summer, the 
Department will host a Superintendent Leadership Forum to help superintendents build 
capacity to implement standards-based education in their districts. The Department 
also awarded grants to all interested education service districts to work with school 
districts in their regions to align local curriculum with content standards, as the first step 
in implementing a standards-based system. 

The Department agrees that school improvement visits consume a great deal of staff 
time but recognizes that will change soon. The governor recently signed Senate Bill 
179, introduced earlier this session at the request of Superintendent Paulus. The bill 
allows annual assurance forms to certify compliance with state standards, rather than 
on-site visits. With the obligation of school improvement visits removed, specialists will 
be able to focus more on building capacity for improving teaching and learning in their 
regions and targeting resources to individual districts in need. 

The Department is working to improve teaching and learning despite significant 
systemic hurdles. In a preliminary analysis conducted earlier this year, the Department 
estimates there are millions of dollars currently in school district budgets for 
professional development. As a result of collective bargaining agreements between 
districts and teachers, the money may be used for activities not necessarily related to 
Oregon’s school improvement plan. The money must be redirected, possibly by 
statute, to activities related to Oregon’s vision for improving teaching and learning. 



4 MAP’s statement that there is no mention of the statewide assessments in the Consolidated District Improvement 
Plan is inaccurate. As stated on page 7 of the Department’s guidelines for Consolidated District Improvement 
Plans, district data analysis focuses on student performance. Student performance is measured by statewide 
assessments. As stated on page 8 of the guidelines, districts are encouraged to develop goals based on improving 
student achievement and performance in content standard areas. Statewide assessments measure student 
achievement and performance in content standard areas. As stated on page 1 1 of the guidelines, districts must 
certify that they are providing programs for students to meet the Certificate of Initial Mastery standards. The 
Certificate of Initial Mastery standards include performance standards on the statewide assessments. 
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Requirements for teacher licensing, certification and continuing education are controlled 
by the Teacher Standards and Practices Commission. Currently, Oregon educators 
need not pursue continuing education after their initial five-year renewal. At the request 
of Superintendent Paulus, Senate Bill 181 was introduced earlier this year. Both it and 
Senate Bill 650 — introduced with Superintendent Paulus’ support — would place the 
Teacher Standards and Practices Commission under the State Board of Education and 
require continuing professional development for license renewal. 

1 0. ODE generally is not viewed as a resource by districts and schools. 

The Department disagrees with this finding. If it is not viewed as a resource by districts 
and schools, it is difficult to understand the average of 250 calls per week received by 
curriculum specialists alone requesting assistance in implementing Oregon’s school 
improvement plan. 5 

The Department recognizes that districts feel pressure when state test results are 
released. The Department is striving to help parents and the general public understand 
that the new standards are quite high and many students will not achieve them 
immediately. It is working with the governor’s office, Oregon Business Council and 
other partners to communicate that information. In releasing the 1996 state test 
results, 6 the Department noted that only 31 percent of tenth graders met the new 
performance standards in mathematics. The public needs to be shown the magnitude 
of the change expected of students. Far from reporting the bad news and washing its 
hands of the matter, the Department developed a multi-pronged strategy to help 
schools improve mathematics teaching and learning. It met with mathematics teachers 
from around the state to gather suggestions for a professional development plan. 
Superintendent Paulus wrote all school boards in the state, describing five actions they 
could take to improve mathematics teaching and learning in their districts. The 
Department organized two mathematics summits for 800 teachers and administrators 
and a mathematics leaders conference for 180 teachers. It located $300,000 in private 
grants for districts to implement the mathematics standards. It focused its Goals 2000 
grants on mathematics and works with higher education to coordinate use of K-16 
Eisenhower professional development money. 

1 1 . Oregon’s ESDs, subject matter associations, universities, and other institutions 
represent vastly underutilized resources available to ODE. 

The Department disagrees with this finding. It regularly collaborates with education 
service districts, subject matter associations, universities, the Northwest Regional 



5 The Department regrets the difficulty MAP experienced reaching two curriculum specialists. The specialists MAP 
traded phone calls with were working in districts that week. 

6 The Oregon Statewide Assessment is not new. Sample assessments of some students began in 1985. Annual 
statewide assessments of all students at the benchmark years began in 1991. 
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Educational Laboratory and other institutions and professional organizations to provide 
technical assistance and professional development. Education service districts have 
worked with the Department on writing assessments since 1991 and mathematics 
problem-solving assessments since 1994. In 1997, more than 120,000 student writing 
samples and more than 120,000 student mathematics problem-solving assessments 
were scored at 17 education service district scoring sites. Education service district 
regional coordinators work closely with schools and districts, acting as Department field 
staff in many ways. Education service districts also sponsor hundreds of staff 
development programs in schools across the state. Two years ago, the Department 
created the Oregon Public Education Network with education service districts. Among 
other things, this network provides online support for K-12 teachers and administrators 
across the state to improve teaching and learning. 

The Northwest Regional Educational Laboratory co-sponsored the mathematics 
summits and provided a wealth of materials for teachers. The Department meets 
weekly with university representatives to align K-12 with college admissions standards. 
The Department held two symposiums last year with professional organizations and 
other partners, resulting in consistent activities, trainings and resources to build 
capacity statewide. In June, the Department will meet with professional organizations 
to draft professional development plans. 

Oregon’s school improvement focuses on improving teaching and learning in six 
curriculum areas, (English, mathematics, science, the social sciences, the arts and 
second languages), six endorsement areas (arts and communications, business and 
management, industrial and engineering systems, health services, natural resources 
and human resources) and career-related learning standards. The Department works 
closely with the professional organizations in these and other areas. The Department’s 
social sciences specialist is outgoing president of the Oregon Council for the Social 
Studies. The Department’s science specialist was president-elect of the Oregon 
Science Teachers Association and continues to be actively involved with that group. 
The president of the Oregon Council of Teachers of Mathematics led a key workshop at 
the recent mathematics summit. The Department’s English and second languages 
specialist works closely with the Oregon Council of Teachers of English and the 
Confederation of Oregon Foreign Language Teachers. The Department also works 
with various other organizations serving disadvantaged and second language learners, 
including the Oregon Summer Bilingual Institute, Oregon Association of Compensatory 
Educators, National Association of State Directors of Migrant Education, Interstate 
Migrant Education Council and Oregon Teachers of English Speakers and Other 
Languages. 

Curriculum associations were essential to developing the academic content standards. 
They reviewed various drafts of the standards and recommended changes. The 
Department often asks curriculum association representatives to present workshops 
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and other professional development activities at events it sponsors and calls on them in 
a variety of other ways to provide subject matter expertise. 7 

1 2. ODE invests little time and few resources in professional development for its own 
employees. 

The Department agrees with this finding. Despite limited financial resources, limited 
out-of-state travel allowed and a heavy press of other demands, staff do manage to 
participate in some professional development activities. There also is significant 
internal collaboration. The Department holds all-staff meetings where the state 
superintendent, associates, assistants and others relate the vision and latest 
developments in Oregon’s school improvement plan. The superintendent meets with 
project managers weekly. Project managers meet every two weeks. Staff in each 
office meet weekly. 

Research and Analysis 

13. As Oregon continues to move to a performance-based system of education, ODE’s 
ability to analyze data will become ever more crucial. However, ODE currently 
possesses little capacity to conduct research and analyze data on key issues such as 
student achievement and school finance. 

The Department agrees that its shift to a standards-based system makes the ability to 
analyze data extraordinarily important. The Department disagrees with the finding that 
it lacks capacity to conduct research and analysis. The information upon which this 
finding is based is inaccurate. The Department never declared the percent of students 
expected to meet the standards. The Department, not the Portland School District, 
analyzed 1996 state test scores and provided results of the study to the district. 8 The 
Department not only is capable of conducting that level of analysis but in fact did so. 9 

The Department agrees that it does not have sufficient data to determine district fiscal 
viability. It is unlikely that the state legislature would cede its analysis of educational 
revenues and expenditures to the Department. 



7 The Department regrets that there were several out-of-date names and phone numbers in its school directory. The 
Department was aware of the problem before the MAP report and continues to work toward solving it. 

8 The correct figures used in the Department’s study of seven school districts are: 2 percent of students who took 
courses below algebra; 8 percent of students who took algebra; 38 percent of students who took geometry; 86 
percent of students who took algebra II; and 100 percent of students who took calculus scored at 239 or above. 

9 It would be unwise to base state policy on a single correlational study. Correlational studies may inappropriately 
link cause with effect. Oregon wants a long-term solution, not a quick fix. In one effort to find long-term solutions, 
professional development research studies are connected with student performance data. 
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14. On the increasingly important dimension of school level information, ODE’s current 
data system is inadequate. 

The Department agrees that it does not collect district expenditure or other district 
financial data. However, it disagrees with the remainder of this finding. 

Data should be collected with a purpose. MAP does not recommend what specific data 
should be collected nor what specific analyses would be possible with the data 
collected. Collecting input data and gauging the success of education based on that 
data is a 1960s model of education. Oregon is concerned with results, not inputs. 

The Department provides the reports described. It provides information to legislators 
regarding how schools across the state are performing. Information regarding how 
Oregon assessment costs compare with similar states also is available. Since 1991, 
the Department has provided district superintendents with reports of how student 
performance in their districts compares with performance in schools with similar student 
populations. The Department provides information to parents comparing performance 
in their children’s schools to performance in all other schools in the state. It also 
provides information to parents and the general public comparing academic 
performance of students in Oregon to students in the United States and other countries. 
It is developing a system to provide information on where high school graduates go to 
college and work. 

The Department disaggregates state test scores by race and ethnic groups and 
socioeconomic factors. Dropout rates are disaggregated by race and ethnic groups. 
The Department also can disaggregate scores by disability category. The Department 
recognizes that a variety of factors combine to produce particular results and routinely 
warns its audiences against viewing data in isolation. 

15. ODE’s current management information system is hobbled by a set of interlocking 
conditions that reduces the utility of the data collected. 

This finding appears to be based in part on an August 1996 report by the manager of 
the Department’s management information system. 10 He is currently working with a 
team of data experts from every office within the Department to develop a better system 
within the confines of fiscal reality. MAP’s cost estimates appear low. 

16. Annual audits represent an underutilized opportunity for appropriate State oversight 
of district expenditure patterns. 

The Department agrees. House Bill 3552, introduced in this legislative session, would 
require performance audits of local districts every five years. House Bill 3553 would 



10 



MAP did not interview the Department’s director. of technology and information resource management. 



require an audit of factors used to compute the State School Fund distribution. 
Expanding local district audits would place an additional cost on local districts. 

Communication 

17. ODE, with limited resources, manages to communicate in multiple ways with policy 
makers, educators, and the public. However, ODE does not currently have a “feedback 
loop” which would enable it to gauge the effectiveness of communication with its 
various "publics.” 

The Department agrees that, with limited resources, it communicates in multiple ways 
with various audiences. It is developing a parent handbook to send to schools by the 
end of this school year. The handbook will address results expected of students and 
other issues of particular concern to parents and their children. The Department 
expects the information will be particularly useful as schools welcome back students 
and parents next fall. 

The Department disagrees with the finding that it lacks a feedback loop. The 
Department seeks feedback in a variety of ways, including but not limited to the 
following: 

• In the last two years, Department staff visited more than 50 percent of the state’s 
schools on school improvement visits, gathering firsthand information from the 
field. Department staff visited many other schools too, including small and rural 
schools to discuss their successes and challenges in implementing the 
Certificate of Advanced Mastery. 

• Educators in the field communicate with their regional contacts. At weekly staff 
meetings, regional contacts report their findings. 

• Department staff attend Principal Leadership Network meetings, Northwest 
Regional Educational Laboratory service provider meetings and monthly district 
superintendent meetings. 

• The 21 st Century Advisory Committee of teachers and administrators appointed 
by the State Board of Education meets regularly to provide input and feedback 
about the Department work. 

• Oregon educators meet regularly with the Department to develop the standards 
and assessment system. 

• Department staff communicate regularly with educators and members of the 
general public via e-mail. The Department maintains a web site and publicizes 
its e-mail and Internet addresses frequently. 

• Focus groups of parents, teachers, business leaders, community representatives 
and others reviewed drafts of the content standards before State Board 
adoption. 

• The Department includes a comment form in many of its school improvement 
publications. 

• The Department periodically includes questions related to school improvement 
in polls conducted by the Oregon School Boards Association and other groups. 
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• The Department regularly reports to and receives feedback from the legislature. 

• The State Board of Education meets regularly with the State Board of Higher 
Education. Department staff meet regularly with higher education staff to ensure 
that the certificate systems align with college admissions requirements. 

II. Recommendations 



Leadership 

1 . The Oregon Department of Education should assume a more activist leadership role 
in improving curriculum and instruction in Oregon’s schools. 

The Department agrees with this recommendation. It agrees with “informed observers” 
who give the Department “generally good marks for.. .carrying out the first tasks in the 
reform plan-building a statewide consensus around the standards....” The Department 
agrees with the recommendation that it should continue to invest its resources in 
developing a technically sound and legally defensible set of assessment instruments 
and procedures. 

2. ODE should be reorganized to reflect a priority for those functions most closely 
related to improving educational programs as well as to enhance necessary 
communication among key program improvement functions. 

The Department agrees that its organizational structure could better reflect its new 
priorities. It also agrees it should add additional staff with subject matter expertise. 
However, adding new staff is difficult at current salary levels. 

The reorganization suggestions described on page 87, section B of the MAP report, 
are less compelling, revisiting the structure in place in 1991." The Department 
disagrees with the recommendation that monitoring and enforcement should be 
delegated to a small group of agency enforcers. Creating an enforcement division 
separate from curriculum and assessment, for example, would deprive the neediest 
schools of the services of those most likely to help. Since 1991, the Department has 
transformed its standardization process into a school improvement process. School 
improvement visits include a monitoring component while shifting most resources to 
review and technical assistance. By statute, on-site visits occur every three years. At 
Superintendent Paulus’ request, the legislature passed Senate Bill 179 to complete the 
shift from compliance monitoring to technical assistance and support by allowing local 
school boards to certify district compliance with standards through annual assurance 
forms. The bill will allow the Department not only to shift state resources to activities 
designed to improve teaching and learning but also to target resources to individual 
district needs. For example, districts with declining test scores, highly mobile student 



11 At that time, the Department had a 21 s1 Century Schools section (capacity building section) and a Standardization 
section (monitoring and enforcement section). 
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populations or numerous citizen complaints would receive assistance specific to their 
needs. 

3. ODE should develop for every organizational unit annual workplans with measurable 
outcomes and budgets specified. Changes to these plans should be negotiated when 
indicated by changing priorities. Managers should be evaluated, at least in part, on the 
basis of how well their performance measures up against approved plans and budgets. 

The Department agrees with this recommendation. Each organizational unit in the 
Department develops annual work plans to guide its activities throughout the year. 
Improvement in student performance is a significant part of each plan. In 1994, the 
Department of Administrative Services implemented a performance management 
process for all management and executive service personnel. This process identifies 
and measures individual, group and organization goals and objectives. Earlier this 
biennium, the Department began implementing a new accounting system. Next 
biennium, the Department plans to strengthen managerial accounting skills and hold 
managers to annual work plans and budget controls. 

4. The Oregon Department of Education, in cooperation with the legislature and 
appropriate state agencies, should take the steps necessary to attract and retain 
education professionals with sufficient expertise, experience, credibility and stature to 
provide the leadership necessary to implement the Educational Act for the 21 st Century. 

The Department agrees it needs to attract and retain education professionals with 
sufficient expertise, experience, credibility and stature. The Department disagrees that 
this would be “cost neutral.” The two-tier specialist classification recommended by MAP 
undoubtedly would be strenuously resisted by union members. 

5. Oregon’s elected position of Superintendent of Public Instruction should be replaced 
with an appointed position. 

The Department agrees with this recommendation. 

Monitoring and Enforcement 

6. Conventional kinds of monitoring and enforcement activities should be streamlined 
and reduced. 

The Department agrees. The change is underway with enactment of Senate Bill 179, in 
particular. 

7. Essential monitoring and enforcement activities should be consolidated into a single, 
relatively small unit of ODE. 
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The Department disagrees with this recommendation for the reasons stated in its 
response to recommendation 2 above. 

The Department is very much aware of EdFlex, a federal initiative to provide school 
districts with flexibility in using federal dollars. EdFlex is based on Oregon’s school 
waiver procedure and was adopted in Congress through the efforts of Oregon’s Senator 
Hatfield. Oregon became the first EdFlex state (and one of only 12 in the country). 
Through EdFlex, the State Board of Education can waive certain federal statutes and 
regulations. The Department also has used federal flexibility provisions in the 
Improving America’s Schools Act and the School-to-Work Act, including provisions 
allowing states and local districts to file consolidated applications for federal funds. 

Capacity Building 

8. The Oregon Department of Education should strategically redeploy the resources 
currently being used for “technical assistance” in Oregon to create multiple networks of 
service providers across the state. The role of ODE should be to serve as a catalyst 
and clearinghouse for new, strategic service provision efforts. 

The Department agrees with this recommendation. In many instances, it does serve as 
a catalyst, convener and organizer of professional development talent in the state and 
connects educators with available services. The Department is moving toward doing 
more in this area. The Department believes there are millions of dollars in existing 
school district budgets for professional development but has no legal authority to 
redirect use of that money. Professional development money the Department does 
control is used for sustained, capacity building activities directed at large groups of 
teachers and administrators who often are trained to return to their school sites and 
train others. 

9. The Oregon Department of Education should assign a much higher priority to the 
professional development of its own staff. 

The Department agrees with this recommendation. At this time, limited financial 
resources, time and out-of-state travel restrictions make it difficult to accomplish this 
goal. 

Research and Analysis 

10. The Oregon Department of Education needs greatly to increase its research and 
analytic capability. 

The Department agrees with this recommendation. 
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1 1. The Oregon Department of Education needs to play a more active role in the 
development, collection and reporting of financial information. 

The Department agrees with the recommendation as stated above. The recommended 
changes would require statutory authority to gather the data suggested. 

12. To enhance its Management Information System, the Oregon Department of 
Education should establish a single database of information that is accessible to 
anyone within ODE and that provides a basis for responsive answers to requests for 
information from outside ODE 

The Department agrees it needs a single database. In fact, the plan to develop such a 
database is described in the Oregon Educational Technology Plan and the Information 
Resources Management Plan. Database development demands financial and human 
resources. Costs may not be entirely recovered. If this was were easy task, many 
state departments of education would have managed it, but few have done so. Arizona 
for example began developing such a database but the legislature refused the $1.5 
million needed. The speed with which the Oregon plan will be implemented is 
contingent on the speed with which the Department locates the necessary resources. 

Some of the data reported to the Department is stored on district computer systems. A 
significant amount of data is not even gathered on district computer systems at present. 
The problem is complex. Hiring an individual to standardize data formats is not enough. 
The Department has hired a database analyst to begin the effort, but the task also 
requires a data base coordinator and systems analyst. 

13. ODE and the State Auditor should expand existing local district CPA audits to 
provide more information for policy makers. 

The Department agrees that this change may be desirable but recognizes it will be an 
additional cost to school districts. House Bill 3553, currently before the legislature, 
would require local district audits of those factors used to compute the State School 
Fund. 

Communication 

14. In promoting a single, unifying vision of what public education is striving to become, 
the Oregon Department of Education should continue to adjust its communications to 
its various “publics” and should develop feedback loops capable of measuring the 
success of its multiple communication efforts. 

The Department agrees with this recommendation. It will continue to use the channels 
of communication identified in the response to finding 17 above and also will develop 
additional means of receiving feedback. 
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There have never been so many public and private partnerships in education as have 
been developed in the last few years. Superintendent Paulus recently signed a formal 
agreement with Oregon’s Chambers of Commerce to provide high school students with 
more opportunities to apply their learning to careers. Through partnerships with the 
Oregon Business Council and Associated Oregon Industries, schools are making their 
curriculum and instruction more rigorous and relevant. The Department recently 
secured grants from Wells Fargo and other private organizations to provide schools 
with funds to meet the new academic content and performance standards. The 
Department created an energy conservation program that saved seven school districts 
$1 .3 million and earned the districts a top national award. It has created numerous 
partnerships with museums and other informal learning centers across the state to 
extend the vast resources of these educational institutions into classrooms. The 
Department is working with the governor’s office, the Oregon Business Council and 
other state education groups on a broad-based, unified approach to education 
improvement. 
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METHODOLOGY 



This study represents a program appraisal of the Oregon Department of 
Education. The scope of this evaluation is ODE's role and function in relation to 
the state's kindergarten through twelfth grade public school system. Specifically 
excluded were the state special schools for the deaf and the blind and the Office of 
Community College Services. The evaluation was conducted by Management 
Analysis and Planning Associates (MAP) a San Francisco-based education 
consulting firm. 

MAP was asked to review the following ODE functions: 1) statutory, 

2) dministration and management, 3) policy making, 4) technical assistance, 

5) onitoring and enforcement, and 6) information management. Study objectives 
included determining ODE's effectiveness in achieving its stated missions, goals, 
and priorities; comparing ODE's mandated functions with its actual activities; 
evaluating ODE's capacity to assess its own performance and the performance of 
Oregon school districts; evaluating management techniques employed by ODE; 
comparing Oregon's Department of Education with relevant alternative models 
from other states; and making recommendations designed to enhance ODE's 
ability to improve teaching and learning at the local district and school levels. 

MAP conducted this evaluation utilizing as an organizing rubric the four 
basic roles of a state education department vis a vis local education agencies: 
providing educational leadership so that a common vision of educational 
excellence is shared throughout the state; building capacity at the local level so 
that schools are better able to carry out that vision; serving as a conduit for state 
and federal monies to the local level; and monitoring for effective, efficient, and 
lawful performance. 
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RESEARCH AND ANALYTIC TECHNIQUES 



In order to gather the most comprehensive and accurate data, MAP 
employed a multi-pronged research strategy which included document analysis; 
interviews; and a telephone survey of a scientifically selected sample of local 
education officials. 

In the first phase of this evaluation, MAP identified the critical issues to be 
explored, developed provisional study questions, and gathered and reviewed 
salient background documents. In addition, preliminary interviews were 
conducted with key policy makers, ODE officials, and other education 
stakeholders. 

Next, MAP refined its workplan, adjusted study questions on the basis of 
results of initial interviews and document analysis, and generated questions for 
the local administrator survey which was conducted as part of this study. In 
developing study questions to guide the course of the evaluation, our criterion 
was: What avenue of investigation is likely to generate information that the 
Legislature and ODE can use to improve teaching and learning in Oregon? 

The third phase of the study consisted of factfinding and analysis. During 
this period, we amassed evidence, through additional document review, 
additional interviews, and the administration of the statewide local educator 
survey, which led to the validation (and sometimes the rejection) of working 
hypotheses. 

Phase four consisted of identifying findings and forming conclusions and 
recommendations. Findings and recommendations were "vetted" with ODE 
officials, other state officials, and with the external advisory board which provided 
advice and counsel to MAP over the course of the study. 

Finally, MAP prepared a final report and submitted it to the client. 
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DOCUMENTS REVIEWED 



MAP reviewed a wide variety of written documents as part of the data 

gathering procedure. An illustrative list of these documents includes: 

• Oregon Education Code 

• Oregon Administrative Rules 

• "Oregon Report Card, 1993-94" 

• "Toward Implementing the Oregon Educational Act for the 21st Century: 
Working Designs for Change" (1993 report of the Oregon State Board of 
Education to the Oregon Legislative Assembly) 

• "Task Force Summaries 1992" (Report of various state level education-related 
task forces) 

• "School Governance and Finance Reform in the Context of Education Reform" 
(June 1992 report of the Task Force on Elementary and Secondary Education) 

• "Questions and Answers" (1995 Oregon Department of Education update on 
the Educational Act for the 21st Century) 

• "Certificate of Advanced Mastery Standards" (August 1996 second draft 
review) 

• "Review of Oregon Content and Performance Standards" (July 1996 Report of 
the National Standards Review Team) 

• "Curriculum Content Framework for Oregon Public Schools" (1994 report) 

• "Oregon Benchmarks: Standards for Measuring Statewide Progress and 
Institutional Performance" (1995 report to the Legislature) 

• "Admission Standards: Content and Process Areas, Proficiencies and 
Indicators" (publication of the Proficiency-Based Admission Standards System 
Project) 

• "Guide to Interpreting the 1995 Oregon Statewide Assessment Results for 
Writing" 
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• A complement of documents distributed to local districts for particular 
financial purposes, including budget worksheets and information on applying 
for a share of Oregon's Goals 2000 dollars 

• "Confederation of Oregon School Administrators and Oregon School Boards 

Association 1995-96 Survey of Salaries and Economic Benefits for 
Administrators in Oregon School Districts and ESDs," Salem, Oregon: 
Confederation of Oregon School Administrators and Oregon School Boards 
Association 

• "1996 Survey of Salaries, Economic Benefits and Selected Policies for Teachers in 

Oregon School Districts," Salem, Oregon: Department of Education 1995-97 
Legislative Adopted Budget Program Narrative. 

• "Oregon's Annual Performance Report: Dwight D. Eisenhower Mathematics 

and Science Education Program," State Education Agency, Program Year 1994- 
95. 

• Working papers of the Oregon Audits Division's 1996 Statewide Audit, 
including apportionment formula calculations. 

INTERVIEWS CONDUCTED 

Interviews are the most efficient means by which to secure first-hand 
information from a wide variety of individuals representing multiple perspectives 
and levels of expertise. In the course of the evaluation of the Oregon Department 
of Education, MAP conducted more than 150 interviews of individuals 
knowledgeable about, or at least keenly interested in, education in the state 
including state officials; Educational Service District (ESD), school district, and 
school level administrators, and representative of key education interest groups. 
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State Level Interviews 



Interviews with individuals whose principal interest or expertise focuses on 
the state level included: 

• State Policy Makers— the Governor, selected (Republican and Democratic) 
members of the Legislature, the Secretary of State, the State Superintendent 
of Public Instruction, and members of the State Board of Education 

• Department of Education Staff— Deputy Superintendent, Office of 
Educational Support Services; Deputy Superintendent, Office of State Board 
Relations; representatives (including individual chiefly responsible for) the 
Office of Assessment and Technology; Office of Curriculum, Instruction, 
and Field Services; Office of Professional Technical Education; Pupil 
Transportation and School District Services; Public Information; School 
Finance and Data Information Services; Offices of Compensatory Education 
and Special Education; and Office of Management Services 

• Representatives of Other Relevant State Agencies — Department of Revenue, 
Office of the State Budget, Legislative Fiscal Office, Teacher Standards and 
Practices Commission, Educational Policy and Planning Office, State 
Chancellor of Higher Education 

• Represen tatives of Major A dvocacy Groups — Confederation of Oregon School 
Administrators, Oregon School Boards Association, Oregon Education 
Association, Oregon Parent-Teacher Association, Oregon Association of 
School Business Officials, and Oregon School Facilities Management 
Association 

• Representatives of Other Education-Affiliated Groups and Organizations — 
Curriculum associations dedicated to mathematics, English/ language arts. 









179 



science, social science, art, foreign language, health, and vocational 
education; the Governor's Quality Education Commission; and the 
Northwest Council for Computer Education 

ESD, District, and School Level Interviews 

Visits were made to, and interviews conducted in, a range of Education 
Service Districts, local school districts, and schools. The purpose of these visits 
and interviews was to gain a first-hand, on-the-ground perspective from "clients" 
regarding their perceptions of the role and function of the Department of 
Education. 

Selected ESD superintendents and other officials were interviewed . 13 At 
the district level, the superintendent and, where possible, other individuals such 
as members of the superintendent's cabinet and district lobbyist were interviewed. 
Selected elementary, middle, and high schools also were visited. On these 
occasions, conversations were held with the principal, often other administrators, 
and with teachers. 

ESD, district, and school visits were designed to provide MAP with 
additional data regarding the ways in which education policy directions 
established at the state level are playing themselves out at the local level. The 
findings from these interviews were further elucidated by the results of the 
survey, described in the next section. 



13 All ESD Superintendents were included in the survey described in the next section of this chapter. 
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THE SURVEY 



A survey of local educators was conducted by Peter Hart Research, under 
contract to MAP for purposes of the ODE evaluation. 

A total of 312 telephone interviews were conducted with school district 
administrators and principals by Hart Research's executive interviewing staff 
between October 17 and November 1, 1996. Of this total, 176 were with local school 
district administrators , 28 were with Education Service District administrators, and 
108 were with principals. In districts whose size required interviews with more than 
one administrator, the superintendent was asked to volunteer the names of one or 
more administrators in the district who had the most contact with the Oregon 
Department of Education. 

After the interviews had been completed, a subsample of the respondents 
was recontacted to verify that the data had been accurately recorded. The 
questionnaires were coded, keypunched, and tabulated on standard computer 
equipment, using Hart Research's in-house facilities. This study is based on a quota 
sample of school district administrators and principals throughout Oregon. The 
sample consisted of a master list of Oregon school districts and ESDs, including their 
superintendents, and a second list of school principals — both provided by the 
Oregon Department of Education. Each list was segmented into five groups based 
on district enrollment, and then stratified geographically. A quota of interviews was 
established for each population group. 

Several approaches were possible in the sampling plan. A typical approach 
would have been to treat each school district administrator as an element, stratify 
the list by school district enrollment, and then interview every "nth" school district 
administrator to attain the total number of interviews, the results, however, would 
have been dominated by superintendents from the very smallest school districts, 
because there are so many of them. 

The approach we chose was to segment the list into five subgroups based on 
student enrollment — large districts with enrollment of greater than 25,000 medium 
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districts with enrollment between 10,000 and 24,999, districts with enrollments 
between 4,000 and 9,999, small districts with between 2,000 and 3,999 students, and 
the smallest districts with enrollments of less than 2,000 students. A quota of 
interviews then was established for each of these five groups, which was roughly 
proportionate to the student enrollment and which would allow for a s uffi cient 
number of interviews for subgroup analysis. Some weighting of the sample by size 
of enrollment was done to reflect more precisely the enrollment pattern within the 
state. 

The reason for drawing a carefully designed, scientific random sample, rather 
than a haphazard sample, was not just to collect a lot of interviews, but to be able to 
make inferences about the attitudes, perceptions, opinions, and characteristics of the 
population and subpopulations from which the sample was 

drawn, and to be able to do so within a known range of what is called "sampling 
error," that is, random variation due to chance, rather than bias. 

Thus, the results are subject to sampling error — i.e. the difference between 
the results obtained from the sample and those that would have been obtained by 
surveying all local district administrators, ESD administrators, and school 
principals. In the case of administrators, the sampling error in this survey is 
minimal, however, because the sample size represents such a large proportion of all 
administrators. This is not quite the case with the principals, as only 108 were 
interviewed statewide. For principals, the results have a margin of error of + 10%. 

STATUTORY REVIEW 

Review of relevant education statutes, rules, regulations, and constitutional 
provisions for purposes of this study was undertaken by Augenblick and Myers 
(A&M) of Denver, Colorado, under contract to MAP. The work began with a 
detailed summary of the Educational Act for the 21 st Century. This statute was 
summarized first because it so directly applied to the scope of the ODE program 
evaluation. 
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Next, remaining statutes were summarized. Review of the education- 
related constitutional provisions and education rules and regulations was also 
undertaken, committed to writing, and shared with the rest of the evaluation 
team. 

A&M's experience in other states, research on policy implementation, and a 
specific frame of reference that comes from a combination of research and 
experience guided this work. John Myers, who led this effort, is a former member 
of the Kansas Legislature and has seven years' experience conducting statutory 
reviews as Education Program Director at the National Conference of State 
Legislatures. 

The research contribution to the analytic frame of reference is derived from 
a number of sources but most particularly from a composite study. Education 
Policy Implementation (edited by Allan Odden, State University of New York 
Press, 1991). Some of the key concepts from the research that framed the statutory 
review were: 1) ability to enforce the statute, 2) level of detail needed in the law, 

3) the connectedness to other policies, and 4) the appropriateness of the policy to 
state level action. 
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ANNOTATED BIBLIOGRAPHY 



Readers interested in delving more deeply into the literature on the 
emerging new leadership role for State Departments of Education in the era of 
standards-based reform are invited to consult these additional sources of 
information 

• "Building Support for Education Reform," Washington, DC: National 
Governors' Association, 1993. 

This report offers a set of strategies for policy makers and educators interested in 
developing effective communication with the public on education reform. 
Suggestions are presented for crafting messages, communicating the same 
message to different audiences, and so on. 

• Corcoran, Thomas B. "Transforming Professional Development for Teachers: 
A Guide for State Policy Makers," Paper prepared for the National Governors' 
Association, New Jersey: Rutgers University, The Consortium for Policy 
Research in Education, October 1994. 

A review of the research regarding successful - and unsuccessful - staff 
development practices. 

• Education Commission of the States, "Bending without Breaking: Improving 
Education through Flexibility and Choice," Denver, CO, June 1996. 

Summarizes state efforts to promote higher standards through increased school 
autonomy. The report also outlines strategies for transitioning from a 
bureaucratic school organization to a more autonomous school structure. 

• Elmore, Richard F. and Susan Furhman, eds. The Governance of Curriculum, 
1994 ASCD Yearbook, ASCD, Alexandria, VA. 

Contains essays on policy development at the state and national levels, state 
curriculum reform development and management and district and school roles in 
curriculum reform by noted researchers and practitioners. 

• Fowler, William J. Jr., "Financial Accounting for Local and State School 
Systems, " 1990. 

The latest version of Handbook 2R2, a school accounting and financial standards 
manual in use in Oregon and many other states. 
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• Fuhrman, Susan, Politics and System Education Reform, Consortium for Policy 
Research in Education, RB-12-04/1994. 

This research report looks at the political problems and challenges facing state 
implementation of system reform. 

• Fullan, Michael G. and Suzanne Stiegelbauer, The New Meaning of 
Educational Change, 1991, Teachers College Press, Columbia University, NY. 

This remains one of the most insightful studies of the complex nature of 
educational change. It contains chapters on the role of state departments of 
education in that process. 

• Mclaughlin, Milbrey and Joan E. Talbert. "Contexts that Matter for Teaching 
and Learning," Stanford University: Center for Research on the Context of 
Secondary School Teaching, March 1993. 

This report highlights major findings from a 5-year nationwide study of various 
education "contexts" - including the structure of professional communities, 
integrated visions of education reform, and school dimensions - that impact 
teaching and learning. 

• "A Motion to Reconsider: Education Governance at the Crossroads," October 
1996, The National Association of State Boards of Education. 

This study reviews the changes and challenges facing state boards of education in 
the fifty states. 

• "A New Architecture for Education Reform," The Business Roundtable, 
prepared by Paul T. Hill and Kelly E. Warner, 1994. 

This report provides a history of the National Business Roundtable's involvement 
in education reform and its proposed guidelines for reform in the fifty states. 

• O'Day, Jennifer, Margaret E. Goertz and Robert E. Floden, "Building Capacity 
for Education Reform, "Consortium for Policy Research in Education, Policy 
Briefs, RB-19-December 1995. 

Describes strategies for increasing school capacity to improve instruction, 
including professional development, assessment and the political process. 
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• "Professional Development for Educators: A Priority for Reaching High 
Standards," Washington, DC: National Governors' Association (no date). 

This very short brochure offers some cogent suggestions regarding the structure of 
professional development and provides a set of questions policy makers should 
ask as they assess the degree to which staff development offerings are useful. 

• Richardson, Virginia (editor). Teacher Change and the Staff Development 
Process, New York: Teachers College Press, 1994. 

A series of case studies on teacher professional development with special 
emphasis on the teaching of reading. 

• Rothstein, Richard, with Karen Hawley Miles, "Where's the Money Gone?," 
Economic Policy Institute, 1995. 

This report argues that school spending has increased less than many observers 
believe over the last two decades and that schools remain relatively productive. 

• Smith, James R., "Leadership Versus Control: A Strategic Approach to Lasting 
School Reform," Middle Grade School State Policy Initiative, Resource Center 
on Educational Equity, Council of Chief State School Officers, Washington, DC, 
1992. 

This monograph directly addresses state education agency officials. It presents a 
three-step model for supporting systemic school reform and offers a series of 
recommendations for effective strategic planning. 

• "The Technology Road Map, A Comprehensive Planning Guide to Computer 
Technology in K-12 School Districts," Microsoft Corporation, 1996. 

A guidebook for forecasting the implementation of technology in schools. 

• "Using What We Have to Get the Schools We Need: A Productivity Focus for 
American Education," A Report by the Consortium on Productivity in the 
Schools, New York: Teachers College, The Institute on Education and the 
Economy, October 1995. 

This report argues that the principal challenge facing American education is to 
make better use of resources. It contains findings and recommendations in the 
areas of governance, management, finance, and teaching and learning. 
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• "What Matters Most: Teaching for America's Future," New York: Report of 
the Nation Commission on Teaching and America's Future, September 1996. 

This report, product of the efforts of a blue ribbon commission composed of 
educators, business leaders and policy makers, presents a comprehensive set of 
recommendations that would revamp teacher preparation over the next decade. 
The report stresses moving to a standards-based system of teacher credentialing 
and intensifying professional development to enable teachers to help students 
meet higher state-established achievement levels. 
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This appendix has been compiled by the Oregon Audits Division in order to 
report two exceptions to Management Analysis and Planning Associates, L.L.C.'s 
(MAP) compliance with generally accepted government auditing standards as 
promulgated by the Comptroller General of the United States in the 1994 Revision 
of Government Auditing Standards. 

During the course of this engagement, the Oregon Audits Division monitored the 
work of MAP to ensure the contractor's compliance with generally accepted 
government auditing standards. This monitoring included reviewing and 
approving the contractor's work plan, reviewing contractor working papers, 
reviewing the contractor's findings and recommendations, reviewing the 
contractor's draft reports, and reviewing the contractor's internal quality control 
system. With the exception of the non-compliance reported below, the Audits 
Division determined MAP's work was conducted in accordance with the 
requirements of the 1994 Revision of Government Auditing Standards. 

The Oregon Audits Division noted two areas of contractor non-compliance with 
auditing standards. These areas are as follows: 

Quality Control. The fourth general standard (Sections 3.31) requires that 
"Each audit organization conducting audits in accordance with these 
standards should have an appropriate internal quality control system in 
place and undergo an external quality control review." 

While MAP's internal quality control system was satisfactory, MAP had not 
undergone an external quality control review; such a review is required 
every three years. However, there were compensating controls for this 
engagement that minimized the risks associated with non-compliance. 
These controls included: 

A. MAP's use of an external peer group of experts in the field of education, 
none of whom were part of the audit team. These experts reviewed MAP' s 
proposal, its workplans, and report drafts, and gave feedback for 
improving each. The peer group consisted of two former state 
superintendents of public instruction, a professor from Stanford University, 
and a former Deputy Superintendent of Public Instruction from the 
California Department of Education. 



B. The State of Oregon's use of a tri-agency committee to oversee the 
contractor and its work. This committee was comprised of two 
representatives from the Legislative Fiscal Office, one representative from 
the Department of Administrative Services, and two representatives from 
the Oregon Audits Division 

C. The Audits Division's review of the contractor's work as cited above. 
The Audits Division is certified by the National State Auditors 
Association's (NSAA) external peer review program. One Audits Division 
member of the tri-agency committee has extensive experience as an NSAA 
peer reviewer, having participated in three peer reviews, both as a peer 
team member and as a peer team supervisor. 

Continuing Education Requirements. The first general standard (Section 
3.6) states that " The staff assigned to conduct the audit should collectively 
possess adequate professional proficiency for the tasks required." Section 
3.6 specifies continuing education and training requirements in terms of 
minimum hours (80) per two year period, no less than 20 of which can be 
attributed to a single year. Twenty-four of the 80 hours must pertain to 
government-related subjects. 

Again, there were compensating controls: 

A. The staff employed on this engagement were highly educated, with five 
of the 10 team members cited on the report cover having doctorates and 
four others having other advanced degrees. 

B. The team members possessed extensive subject matter expertise. 

As a result of the foregoing analysis, the Oregon Audits Division concluded that 
the non-compliance cited above had no adverse impact on the quality of the 
contractor's work or on the quality of the final report. 



Facts About the Secretary of State Audits Division 



The mission of the Audits Division is to "Protect the Public Interest and Improve 
Oregon Government.” The Oregon Constitution provides that the Secretary of State 
shall be, by virtue of his office. Auditor of Public Accounts. The Audits Division exists 
to carry out this duty. The division reports to the elected Secretary of State and is 
independent of the Executive, Legislative, and Judicial branches of Oregon government. 
The division audits all state officers, agencies, boards, and commissions and oversees 
audits and financial reporting for local governments. 



Directory of Key Officials 

Acting State Auditor 
Deputy State Auditor 



Sam Cochran 

Sharron Goggins, CPA, CFE 



This report is intended to promote 
the best possible management of public resources. 




Oregon Audits Division 
Public Service Building 
Salem, Oregon 97310 
503-986-2255 Hotline: 800-336-8218 
Internet: Audits .Hotline@state. or. us 
http ://www.sos .state, or. us/ audits/ audithp .htm 



If you received a copy of an audit and you no longer need it, you may return it to 
the Audits Division. We maintain an inventory of past audit reports, and your 
cooperation will help us save on printing costs. 

We invite comments on our reports through our Hotline or Internet address. 



Audit Cost: $280,000 

Percent of Program Expenditures: Less than 1% 

Costs Savings/Revenue Enhancements: Streaml i ning 
and consolidating essential monitoring and 
enforcement activities will result in undetermined 
savings. 
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